Does a mentoring program for new peer reviewers improve their review quality? a randomized controlled trial

Abstract

Objective Traditional methods of peer reviewer training and selection have been shown to be ineffective. We instituted a mentoring program, pairing senior reviewers (selected for high quality) and new reviewers on the same manuscripts, to see if this intervention would improve quality as measured by editors' validated quality scores of all reviews. Design Over a 2-year period, all new reviewers were randomly assigned to a control group or an intervention group by blinded technique. The intervention group was invited by e-mail to join our mentoring program and asked to communicate with their assigned senior reviewer mentor (by e-mail or phone) each time they were assigned a manuscript. Mentors were volunteers chosen for consistent timeliness and quality over years. Mentors and mentees (who were paired by topic during the study) were also notified each time either was assigned a manuscript to review; both groups (and editors) were blinded as to the study intervention. The content and amount of communication were left to the mentor and mentee. After 3 reviews, mentees were surveyed regarding their experience. We calculated reviewer-specific and average trend lines for both groups, accommodating for lack of independence of ratings using SAS Proc MIXED models (ver 9.1, SAS Institute, Cary, NC) with a random intercepts and slopes correlation structures. We tested for longitudinal differences in the average trends differences between the 2 groups during the study period. Results A total of 17 mentees, 15 controls, and 16 mentors completed 194 reviews. Both mentees and control group reviewers received the same number of invitations, but mentees accepted and completed more reviews than control group reviewers (109 vs 84), and mentee mean scores were higher than control group scores when controlling for within reviewer trends and variations in volume and group trend effects (3.81 vs 3.24; difference: –0.56 [95% confidence interval, –1.048 to –0.078], P =.027). Satisfaction was not assessed since it has previously been shown not to predict performance, but participants were surveyed for their suggestions. Conclusion A simple system of pairing newly recruited peer reviewers with volunteer reviewer mentors chosen for consistent quality over time resulted in slightly more reviews accepted and higher review scores.