A survey of methods for improving review quality

Abstract

For peer review to be successful, students need to submit high-quality reviews of each other’s work. This requires a certain amount of training and guidance by the review system. We consider four methods for improving review quality: calibration, reputation systems, meta-reviewing, and automated meta-reviewing. Calibration is training to help a reviewer match the scores given by the instructor. Reputation systems determine how well each reviewer’s scores track scores assigned by other reviewers. Meta-reviewing means evaluating the quality of a review; this can be done either by a human or by software. Combining these strategies effectively is a topic for future research.