Observed score reliability indices in diagnostic classification models

Abstract

Quantifying the reliability of latent variable estimates in diagnostic classification models has been a difficult topic, complicated by the classification-based nature of these models. In this study, we derive observed score reliability indices based on diagnostic classification models as an extension of classical test theory-based reliability. Additionally, we derive conditional observed sum- and sub-score distributions. In this manner, various conditional expectations and conditional standard error of measurement estimates can be calculated for both sum- and sub-scores of a test. The proposed methods provide a variety of expectations and standard errors for attribute estimates, which we demonstrate with an analysis of an empirical test. Moreover, a simulation study revealed the proposed sub-score-based reliability index was correlated to a previously developed attribute mastery reliability index.