USING PAST DATA TO ENHANCE SMALL-SAMPLE DIF ESTIMATION: A BAYESIAN APPROACH

Abstract

The application of the Mantel-Haenszel test statistic (and other popular DIF-detection methods) to determine DIF requires large samples, but test administrators often need to detect DIF with small samples. There is no universally agreed upon statistical approach for performing DIF analysis with small samples; hence there is substantial scope of further work on the problem. One advantage of a Bayesian approach over a frequentist approach is that the former can incorporate, in the form of a prior distribution, existing information on the inference problem at hand; a prior distribution often leads to improved estimation, especially for small samples. Further, for any operational test, a huge volume of past data is available, and for any item appearing in a present test, there is a high chance that a number of similar items have appeared on past operational administrations of the test. Therefore, ideally, it will be possible to use that past information as a prior distribution in a Bayesian DIF analysis. This paper discusses how to perform such an analysis. The suggested Bayesian DIF analysis method is shown to be an improvement over the existing methods in a realistic simulation study.