Exploring reviewer self-assessment in the context of academic peer review

Abstract

Scientific journals often rely on informal methods to evaluate reviewers, such as editor ratings and author feedback. Reviewer self-assessment offers a promising, yet underexplored, approach to improving the peer-review process. This study examined the factors associated with reviewers’ self-assessments. We surveyed 642 reviewers and editors from three Information Systems (IS) conferences (January–February 2020), and 144 responses were analyzed using quantitative inferential statistics. Most respondents were male (72.2%) and based in Europe (59%). We found no significant association between self-assessment and conventional experience markers (reviewing and publishing experiences). In contrast, significant associations were observed between higher self-assessment and the perceived importance of feedback from editors (χ2 = 19.689, p ≈ 0.002), feedback from authors (χ2 = 25.168, p < 0.001), and formal training (χ2 = 14.64, p ≈ 0.047). Although our sample comes from IS settings, these mechanisms are process-based; therefore, these findings could be extended to a broader peer review ecosystem. Overall, organizational interventions, structured feedback from editors and authors, and formal training are more closely related to reviewers’ self-assessments than accumulated publishing or reviewing experience.