Peer review for improving the quality of grant applications

Abstract

Background: Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce. Objectives: To estimate the effect of grant giving peer review processes on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research. Search strategy: Electronic database searches and citation searches; researchers in the field were contacted. Selection criteria: Prospective or retrospective comparative studies with two or more comparison groups assessing different interventions or one intervention against doing nothing. Interventions may regard different ways of screening, assigning or masking submissions, different ways of eliciting opinions or different decision making procedures. Only original research proposals and quality outcome measures were considered. Data collection and analysis: Studies were read, classified and described according to their design and study question. No quantitative analysis was performed. Main results: Ten studies were included. Two studies assessed the effect of different ways of screening submissions, one study compared open versus blinded peer review and three studies assessed the effect of different decision making procedures. Four studies considered agreement of the results of peer review processes as the outcome measure. Screening procedures appear to have little effect on the result of the peer review process. Open peer reviewers behave differently from blinded ones. Studies on decision-making procedures gave conflicting results. Agreement among reviewers and between different ways of assigning proposals or eliciting opinions was usually high. Authors' conclusions: There is little empirical evidence on the effects of grant giving peer review. No studies assessing the impact of peer review on the quality of funded research are presently available. Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile. Copyright © 2008 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd.