The Affordances of Multivariate Elo-Based Learner Modeling in Game-Based Assessment

Abstract

Previous research and experiences have indicated the potential that games have in educational settings. One of the possible uses of games in education is as game-based assessments (GBA), using game tasks to generate evidence about skills and content knowledge that can be valuable. There are different approaches in the literature to implement the assessment machinery of these GBA, all of them having strengths and drawbacks. In this article, we propose using multivariate Elo-based learner modeling, as we believe it has a strong potential in the context of GBA for three aims: first, to simultaneously measure students competence across several knowledge components in a game; second, to predict task performance; and finally, to estimate task difficulty within the game. To do so, we present our GBA Shadowspect, which is focused on solving geometry puzzles, and we depict our implementation using data collected from several high schools across the USA. We obtain high-performing results (AUC of 0.87) and demonstrate that the model enables analysis of how each student's competency evolves after each puzzle attempt. Moreover, the model provides accurate estimations of each task's difficulty, enabling iterative improvement of the game design. This study highlights the potential that multivariate Elo-based learner modeling has within the context of GBA, sharing lessons learned, and encouraging future researchers in the field to consider this algorithm to build their assessment machinery.