The JBJS Peer-Review Scoring Scale: A valid, reliable instrument for measuring the quality of peer review reports

Abstract

Many journals seek to evaluate the quality of reviews performed by their panel of reviewers. The purpose of this study is to determine if members of a journal editorial board can consistently and reliably use a single numeric scoring system to evaluate the quality of peer reviews. A retrospective analysis of 11 randomly selected manuscripts that had undergone external peer review by three reviewers was performed. Six had been rejected and five accepted. Each deputy editor was asked to score each of the reviews. The intraclass correlation was determined for each of the manuscripts to determine the consistency in grading. The intraclass correlation for 10 of the 11 manuscripts was above 0.87. This study demonstrates that an editorial board of deputy editors, without external training, can consistently and reliably grade reviews with excellent agreement.