When Cut Scores Impact Human Ratings: An Extended Many-Facet Rasch Model for Category-Specific Rater Severity Shifts

Abstract

The many-facet Rasch model (MFRM), widely used to analyze and evaluate rater-mediated assessments, focuses on between-rater differences in overall severity or leniency across facets and rating scale categories. Studying within-rater severity differences, particularly abrupt shifts in a rater's tendency to assign harsh or lenient ratings around specific scale categories, requires an extended modeling approach. Specifically, to focus on local, category-dependent severity effects, we propose the “many-facet Rasch model for category-specific severity shifts” (MFRM-CSS). Building on Bayesian parameter estimation, we demonstrate the model's suitability for examining local severity changes at scale categories with special significance as cut scores. In a simulation study and real-data analysis, we found that (a) differences in category-dependent severity levels affected observed score distributions and passing rates, (b) the MFRM-CSS reliably recovered true overall and local rater severity parameters, (c) ignoring local severity shifts biased overall severity and category threshold estimates, and (d) the MFRM-CSS outperformed the baseline MFRM, which does not account for category-specific severity shifts, in data-model fit when applied to essay rating data. In the real dataset, the greatest impact of local severity was observed at the second threshold, where the pass-fail cut score was set. The discussion highlights the practical implications of applying the MFRM-CSS and suggests directions for future research.