SIMULATION STUDIES APPLYING POSTERIOR PREDICTIVE MODEL CHECKING FOR ASSESSING FIT OF THE COMMON ITEM RESPONSE THEORY MODELS

Abstract

Model checking in item response theory (IRT) is an underdeveloped area. No universal model-checking tool exists in this field. The posterior predictive model-checking method (Guttman, 1967; Rubin, 1981, 1984) is a popular Bayesian model-checking tool because of its simplicity, strong theoretical basis, obvious intuitive appeal, and ability to provide graphical evidence. This study applies the posterior predictive model-checking method to assess the fit of the popular IRT models (Rasch model, 2PL model, and 3PL model) for tests with dichotomous items. A series of simulation studies applying the method yields promising results. An important issue with the application of the posterior predictive model-checking method is the choice of discrepancy measures (which are like the test statistics in classical hypothesis testing). This article examines the performance of a number of discrepancy measures for assessing different aspects of fit of the common IRT models and suggests simple summarization of the results. Simple graphical summaries provide useful insight about the fit of the models. The odds ratios, corresponding to the responses of the examinees to pairs of items, appear to be powerful discrepancy measures in detecting several types of misfits of the IRT models. The observed score distribution and biserial correlation coefficients are also found useful as discrepancy measures on a number of occasions. A companion article applying the posterior predictive method to real data examples shows that the method has the potential to become very useful in psychometrics.