Finding an optimal balance between agreement and performance in an online reciprocal peer evaluation system

Abstract

Consistent with movements integrating performance evaluation and improvement, the current study examines an assumption that more peer raters will produce better results or the so-called maxima strategy. This study examines the maxima strategy from both the agreement and performance perspectives with the intent of examining the role of feedback information in reliability and performance through an optimal number of peer raters per student ratee. It was found that the maxima strategy works consistently from agreement perspectives, whereas the relationship between the maxima strategy and student performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of peer raters needs to be decided according to the optimal balance between reliability and performance, which maximizes ratees’ performances without sacrificing evaluation reliability.