Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists

Abstract

Background: A 'societal impact factor' that complements the scientific impact factor would contribute to a more comprehensive evaluation of scientific research. In order to develop a practical tool for its assessment, it is important to learn about perceptions of scientists on how to measure a societal impact factor. Methods. This qualitative study presents the development of a practical tool to measure the societal impact of publications based on 8 focus group discussions with 24 biomedical scientists at the Medical University Vienna between May 2008 and May 2009. Topics focused on (1) features of an ideal tool, (2) criteria that should be considered in the assessment, and (3) the identification of practical pitfalls. In an iterative exercise involving the repeated application of the drafted tool to scientific papers, criteria for the assessment were refined. A small-scale exercise to evaluate the tool in terms of its comprehensibility, relevance and practicability was conducted using questionnaires for 6 external experts in leading positions of public health, and yielded acceptable results. Results: The tool developed consists of three quantitative dimensions, that is (1) the aim of a publication, (2) the efforts of the authors to translate their research results, and, if translation was accomplished, (3) (a) the size of the area where translation was accomplished (regional, national or international), (b) its status (preliminary versus permanent) and (c) the target group of the translation (individuals, subgroup of population, total population). Conclusions: Focus group discussions with scientists suggested that the societal impact factor of a publication should consider the effect of the publication in a wide set of non-scientific areas, but also the motivation behind the publication, and efforts by the authors to translate their findings. The proposed tool provides some valuable insights for further research and practical applications in the topic area. © 2011 Niederkrotenthaler et al; licensee BioMed Central Ltd.