Computerized adaptive test using raw responses for item selection: Theoretical results and applications for the up-and-down method

Abstract

Modern computerized adaptive testing (CAT) is finding applications that contain more intensive assessments, collected over nontraditional devices such as tablets and smartphones. In this paper, we introduce an CAT algorithm that uses raw responses to adaptively select items and does not require updating the ability estimate at every administration of an item. The proposed algorithm is especially useful in adaptive assessment situations in which updating ability estimate at each administration is either not feasible or too costly to implement. Specifically, an a-stratified multistage up-and-down method is proposed as an approximation to the commonly used recursive maximum likelihood estimate (R-MLE). Using Markov chain tools, we derive theoretical results for the statistical properties of the upand-down method. We also report empirical studies for the performance of the proposed method. Both simulation experiments and real data analysis are included. Limitations of the method such as reduced statistical efficiency are also discussed. Overall, despite the limitations, our results show that the up-and-down method is a promising alternative to the classical R-MLE and well-suited for some CAT applications such as ecological momentary assessments.