Using institutional data to evaluate game-based instructional designs: Challenges and recommendations

Abstract

Over the course of the last 5 years, an iterative attempt has been made to develop a series of alternate reality learning games to support undergraduate students in an introductory computer literacy course. During that time, three separate narratives and 18 different iterations of the course have evolved in response to quantitative assessment and course evaluation data as well as qualitative data captured in reflective student web log reflections and interviews with instructors and learners. One major challenge to completing assessments of each iteration was the dearth in the availability of institutional data such as demographics including student year classification or student major (i.e., freshman/business major). Lacking such data that is necessary to conducting factor and other forms of statistical analysis, we have developed approaches during that time to capture necessary information in order to more accurately assess and evaluate the effectiveness of the learning game components as well as the degree to which satisfaction levels could be detected. This chapter discusses both our challenges in using institutional data for research on learning games at the postsecondary level and suggests, from our experience, means to overcome these using technological and methodological approaches.