Measurement Error and Bias in Value‐Added Models

Abstract

By aggregating residual gain scores (the differences between each student's current score and a predicted score based on prior performance) for a school or a teacher, value‐added models ( VAMs ) can be used to generate estimates of school or teacher effects. It is known that random errors in the prior scores will introduce bias into predictions of the current scores, and thereby, into the estimated residual gain scores and VAM scores. The analyses in this paper examine the origins of this bias and its potential impact and indicate that the bias is an increasing linear function of the student's prior achievement and can be quite large (e.g., half a true‐score standard deviation) for very low‐scoring and high‐scoring students. To the extent that students with relatively low or high prior scores are clustered in particular classes and schools, the student‐level bias will tend to generate bias in VAM estimates of teacher and school effects. Adjusting for this bias is possible, but it requires estimates of generalizability (or reliability) coefficients that are more accurate and precise than those that are generally available for standardized achievement tests. Report Number: ETS RR‐17–25