Detection of unusual test administrations using a linear mixed effects model

Abstract

With an increase in the number of administrations of test forms, there is also an increase in the complexity of the quality assurance procedures needed to maintain the stability of the reported scores. Many traditional methods for quality control (QC; Allalouf, Educational Measurement: Issues and Practice 26(1):36–46, 2007) are not sufficient for detecting unusual scores in a rapid flow of administrations, for which new QC approaches have been proposed in recent years. In this study we investigate data from 15 consecutive administrations that follow a specific equating design (or braiding plan) and propose a linear mixed effects model for the detection of abnormal results. In this project we investigate the effects of the braiding plans and estimate the effects of several identified factors that influence the means and variances of the scale scores. This analysis is appropriate for tests with a continuous or almost continuous administration mode. The data are from a global standardized assessment of English skills. Reading and Listening sections of the English test were modeled separately. Test-takers’ education level, major, years abroad, years of study, and whether they repeated the test turned out to have significant effects on the scores. In addition, a formula for a prediction interval for the scaled mean score of certain subgroups was proposed to detect unusual administrations.