The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment

Abstract

Our society currently relies heavily on test scores to measure individual progress, but typical scores can only provide a limited amount of information. For instance, a test score does not reveal which of the assessed topics were mastered and which were not well understood. According to the U.S. government, this is no longer sufficient. The No Child Left Behind Act of 2001 calls for diagnostic information to be provided for each individual student, along with information for the parents, teachers, and principals to use in addressing individual student needs. This opens the door for a new area of psychometrics that focuses on the inclusion of diagnostic feedback in traditional standardized testing. This diagnostic assessment could even be combined with techniques already developed in the arena of computer adaptive testing to individualize the assessment process and provide immediate feedback to individual students. This dissertation is comprised of two major components. First, a cognitive diagnosis-based model, namely the fusion model, is applied to two large-scale mandated tests administered by the Texas Education Agency; and secondly, computer adaptive testing technology is incorporated into the diagnostic assessment process as a way to develop a method of providing interactive assessment and feedback for individual examinees' mastery levels of the cognitive skills of interest. The first part requires attribute assignment of the standardized test items and the simultaneous IRT-based estimation of both the item parameters and the examinee variables under the fusion model. Examinees are classified dichotomously into mastery and non-mastery categories for the assigned attributes. Given this information, it is possible to identify the attributes with which a given student needs additional help. Results from the first portion indicate that the fusion model is indeed an appropriate approach to cognitive diagnosis in a real large-scale assessment. The second part focuses on applying CAT-based methodology, and in particular item selection, to the diagnostic testing process to form a dynamic test that is sensitive to individual response patterns while the examinee is being administered the test. This combination of computer adaptive testing with diagnostic testing will contribute to the research field by enhancing the results that students and their parents and teachers receive from educational measurement. Results from this second portion of this dissertation indicate that item selection based on both the overall score and the diagnostic attribute pattern is comparable to item selection based solely on the overall score and is better than selecting items based solely on the diagnostic attribute pattern.