Using the Rasch model to quantify the causal effect of test instructions

Abstract

This experimental study was designed to quantify, by means of the Rasch model (RM), the effects of three instruction/scoring conditions on student measures and on the reliability of an achievement multiple-choice test in a field context. Examinees performed the test in one of three conditions which differed only in the instructions provided. Predictions regarding performance indicators were fulfilled, and the expected differences in reliability favoring omission-inducing instructions did appear. This difference in reliability was found for both Rasch and raw data and thus it can be concluded that the fact that results from previous studies failed to corroborate this prediction must have been due to the lack of important consequences of test scores for the students. The RM has served to neatly quantify the differences between instructions promoting guessing and instructions promoting omission under uncertainty, showing that the recommendation to omit is not only educationally but also psychometrically sound. Copyright 2007 Psychonomic Society, Inc.