Vertical comparison using reference sets

Abstract

When tests for different populations are compared, vertical item response theory (IRT) linking procedures can be used. However, the validity of the linking might be compromised when items in the procedure show differential item functioning (DIF), violating the assumption of the procedure that the item parameters are stable in different populations. This article presents a procedure that is robust against DIF but also exploits the advantages of IRT linking. This procedure, called comparisons using reference sets, is a variation of the scaling test design. Using reference sets, an anchor test is administered in all populations of interest. Subsequently, different IRT scales are estimated for each population separately. To link an operational test to the reference sets, a sample of the items from the reference set is administered with the operational test. In this article, a simulation study is presented to compare a linking method using reference sets with a linking method using a direct anchor. From the simulation study, we can conclude that the procedure using reference sets has an advantage over other vertical linking procedures.