A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or closed peer review models

Abstract

Background / Purpose: The objective of this study was to assess i) whether reports from reviewers recommended by authors show a bias in quality and recommendation for final decision, compared with reviewers suggested by editors, and ii) whether reviewer reports for journals operating on open or closed peer review models differ.We compared two journals of similar sizes and rejection rates, BMC Microbiology and BMC Infectious Diseases. BMC Microbiology uses single-blind peer review, and BMC Infectious Diseases open peer review. All the procedures for handling manuscripts and peer review are identical in the two journals, except for the reviewers being anonymous in the biology journal and being named in the medical journal.In each journal we analysed 100 manuscripts that had a final decision (accept or reject). Each manuscript had two reviewers, one suggested by the authors and one by another party. Each reviewer report was rated using an established Review Quality Instrument (1). Main conclusion: There was no difference in the quality of reports between author- and editor-suggested reviewers. There was, however, a difference in the overall quality of reports between the open and closed peer review journals. The overall score was 5% higher under the open model, owing mainly to higher scores on questions relating to feedback on the methods (11% higher), constructiveness (5% higher), and amount of evidence substantiating reviewers comments (9% higher).Although reviewers suggested by authors are more likely to recommend acceptance, editors appear to acknowledge this potential bias and put more weight on the reports from the other reviewer. The quality of reports is higher on the open peer review model.Copyrights: Table two was reproduced with kind permission of van Rooyen S et al. (1999) BMJ 1999 Jan 2 318(7175); 23-7.