Effects of experimental interventions to improve the biomedical peer-review process: A systematic review and meta-analysis

Abstract

BACKGROUND: Quality of the peer-review process has been tested only in small studies. We describe and summarize the randomized trials that investigated interventions aimed at improving peer-review process of biomedical manuscripts. METHODS AND RESULTS: All randomized trials comparing different peer-review interventions at author-, reviewer-, and/or editor-level were included. Differences between traditional and intervention-modified peer-review processes were pooled as stand-ardized mean difference (SMD) in quality based on the definitions used in the individual studies. Main outcomes assessed were quality and duration of the peer-review process. Five-hundred and seventy-five studies were retrieved, eventually yielding 24 randomized trials. Eight studies evaluated the effect of interventions at author-level, 16 at reviewer-level, and 3 at editor-level. Three studies investigated interventions at multiple levels. The effects of the interventions were reported as mean change in review quality, duration of the peer-review process, acceptance/rejection rate, manuscript quality, and number of errors detected in 13, 11, 5, 4, and 3 studies, respectively. At network meta-analysis, reviewer-level interventions were associated with a significant improvement in review quality (SMD, 0.20 [0.06 to 0.33]), at the cost of increased duration of the review process (SMD, 0.15 [0.01 to 0.29]), except for reviewer blinding. Author-and editor-level interventions did not significantly impact peer-review quality and duration (respectively, SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for quality, and SMD, 0.17 [−0.16 to 0.51] and SMD, 0.19 [−0.40 to 0.79] for duration). CONCLUSIONS: Modifications of the traditional peer-review process at reviewer-level are associated with improved quality, at the price of longer duration. Further studies are needed. REGISTRATION: URL: https://www.crd.york.ac.uk/prospero; Unique identifier: CRD42020187910.