Evaluating the impact of a scientific institution by use of free web tools

Abstract

Introduction: The Belgian Health Care Knowledge Center (KCE) is a federal institution established in 2003 that produces studies and reports to advice policy-makers about health care and health insurance.

Aims: To describe a model of tools that allows to look at what is said about an organisation on the web and social media and expose the experience gained after more than one year of testing.

Methods: A general model of monitoring has been defined and produced with a selection of free web tools as modular elements. In a second step, queries have been created to monitor the impact of the institution, or a specific publication. Identified items are curate by the librarian for dissemination to the patrons through various channels. Use and utility of the curated items have been evaluated through a survey of patrons.

Discussion: After one year of operations, the system allowed to identify 924 items related to our institution or its scientific reports; from them 229 were not already identified through a standard press review. The communication manager at the origin of the project was very pleased with these results.
They permitted to identify unexpected groups debating on our reports; also false information was identified, and corrected by the editors. The selection of information sources and the tool’s maintenance requires a significant time investment. In addition, the use of free tools,
that can be seen as an advantage, must be weighed against a possible lack of continuity or maintenance.
However, failing applications can be replaced by an equivalent; but not necessarily with exactly the same options.
The three types of audiences targeted by the tool does not valued it in the same way. The management accesses to the information but items are rarely useful. For the researchers the information proposed is interesting but rarely useful in their daily work environment. The staff of the institution in its majority doesn’t consult it.

Conclusion: The modular structure has proven its effectiveness, but the workload is high and the benefits for customers are not always obvious. In the specific context of our institution, the balance cost / benefit is not fully visible and should lead to assess the interest of maintaining these tools. But from a broader point of view, free tools helped to test the need without having to buy costly professional tool.