Comparison of DIF Detecting Methods in Cognitive Diagnostic Test

Abstract

DIF detecting is an important issue when using cognitive diagnostic tests in practice. MH method, SIBTEST method and Wald test have been introduced into cognitive diagnostic test DIF detection. However, all of them have some limitations. As Logistic Regression is not based on certain model, has a pretty good performance in detecting DIF in IRT test, and could distinguish uniform DIF from non-uniform ones, it is predictable that Logistic Regression (LR) could make up some of the flaws the methods used in cognitive diagnostic test have. The performance of LR was compared with that of MH method and Wald test. Matching criteria, DIE type, DIF size and sample size were also considered. In this simulation study, data was generated using HO-DINA model. When detecting DIF using MH method and LR, 3 kinds of matching criteria were used. Sum score was computed by summing up right answers of each examinee; θ was calculated with 2PL model; KS was calculated with 3 different cognitive diagnostic methods, which are DINA model, RSM method and AHM method. Wald test could be directly applied to DINA model. The 4 kinds of DIE are s increases, g increases, s and g increase simultaneously, and s increases while g decreases. Two levels of DIE size are 0.05 and 0.1. Two levels of sample size are 500 examinees per group and 1000 examinees per group. Here are the results: (1) LR did a great job in cognitive diagnostic test DIF detection with a pretty high power and low type I error; (2) LR was not constrained by cognitive diagnostic models, thus it can use KS estimated by whichever cognitive diagnostic methods; (3) LR can distinguish uniform DIF from non-uniform DIF, and the power and type I error are fairly good; (4) Using KS as matching criteria in cognitive diagnostic test DIF detection can provide ideal power and type 1 error; (5) With the increase of DIE size and sample size, power grew significantly while type I error rate did not change. LR has a satisfying performance in cognitive diagnostic test DIF detection with a high power and a low and stable type 1 error rate. KS should be the ideal matching criteria in cognitive diagnostic test DIE detection. In the long run, the unique characters of DIF in cognitive diagnostic test should be explored, and pertinence DIF detecting methods should be developed. (PsycINFO Database Record (c) 2016 APA, all rights reserved)