Beyond normalized gain: Improved comparison of physics educational outcomes

Abstract

This study proposes methods of reporting results of physics conceptual evaluations that more fully characterize the range of outcomes experienced by students with differing levels of prior preparation, allowing for more meaningful comparison of the outcomes of educational interventions within and across institutions. Factors leading to variation in post-test scores on the Force and Motion Conceptual Evaluation (FMCE) across different instructors, semesters, and course models in a sample collected in introductory calculus-based mechanics at a large, eastern land-grant university were examined. The sample was collected over nine years and contains a total of N=4409 matched pretest and post-test records. The data showed a systematic semester-by-semester variation in both pretest scores and ACT or SAT mathematics percentile scores. Neither the normalized gain nor Cohen's d removed the semester-to-semester variation observed in post-test scores. The local average curve plotting post-test scores against pretest scores, which we call a conceptual growth curve, allowed for the characterization of outcomes for students with different pretest scores. Regression models were used to produce an approximation to this curve. By using either the full curve or a mathematical approximation developed through linear regression, the post-test score that would be observed if a class enrolled students with a given level of prior preparation measured by pretest scores can be predicted. This predicted post-test score can then be used to calculate the predicted normalized gain if desired. These methods rely on using the natural variation of incoming student preparation at one institution to predict how a class would perform if it enrolled students with different prior preparation. The study presents an example of converting the outcomes at an institution with a weakly prepared student population to the outcomes which would have been observed if the course enrolled a more prepared student population; converting the outcomes for a different student population dramatically changed the interpretation of how the class studied was functioning.