Examination of quantitative methods for analyzing data from concept inventories

Abstract

Physics education research (PER) has long used concept inventories to investigate student learning over time and to compare performance across various student subpopulations. PER has traditionally used normalized gain to explore these questions but has begun to use established methods from other fields, including Cohen's d, multiple linear regression, and linear mixed effects models. The choice of analysis method for examining student learning gains in PER is a current subject of debate. We synthesize this debate here by focusing on the research questions and interpretations that can be drawn using various statistical tools. We focus particularly on how results can be interpreted through an equity lens and what questions can and cannot be answered using different methods. Our results demonstrate the importance of clearly defining a research question and using appropriate tools to answer that question.