A polytomous item response model that simultaneously considers bias factors of raters and examinees: Estimation through a Markov Chain Monte Carlo algorithm

Abstract

It is generally known that evaluation of abilities through essay tests, interviews, and performance assessments may entail both rater biases, such as severity, dispersion of scores, and daily fluctuations, and examinee biases, such as expectation effects, order effects, and beauty of handwriting. In the present article, an item response model is proposed for such data, based on the Generalized Partial Credit Model (GPCM; Muraki, 1992) for polytomous responses. Effects of rater and examinee biases can be estimated directly and simultaneously through the proposed model. Parameter estimation was performed via the Markov Chain Monte Carlo (MCMC) method, which is becoming acknowledged as an effective tool for item response models. A simulation study indicated stable convergence of estimates. Additionally, actual essay test data, in which 4 raters evaluated the essays written by 304 high school students, were analyzed; the results showed the efficacy of the proposed model.