How to quantify the efficiency of a pedagogical intervention with a single question

Abstract

In many situations, the change in the conceptual understanding of students is measured using a single question. This is, for instance, the case in peer instruction where students answer twice to the same questions, before and after the discussion phase. Using item response theory and assuming that students proficiencies are normally distributed, it is shown that the Cohen's d effect size characterizing the change of mean proficiencies can be estimated by taking 0.6 times the log of the odds ratio of class scores. Moreover the polychoric correlation coefficient between students' answers is suggested as an additional indicator to detect abnormal changes in scores when its value is below 0.3. Taken together, these two indicators give both a precise measurement of a pedagogical intervention - a peer discussion or something else - and a coefficient of security to detect random answers or poor writing of questions. The application is made to the evaluation of peer discussions that took place in an introductory mechanics course taught using peer instruction.