Reviewing the review: a qualitative assessment of the peer review process in surgical journals

Abstract

Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions. Manuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher’s exact tests. Impact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance. Substantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.