Logs or Self-Reports? Misalignment between Behavioral Trace Data and Surveys When Modeling Learner Achievement Goal Orientation

Abstract

While learning analytics researchers have been diligently integrating trace log data into their studies, learners' achievement goals are still predominantly measured by self-reported surveys. This study investigated the properties of trace data and survey data as representations of achievement goals. Through the lens of goal complex theory, we generated achievement goal clusters using latent variable mixture modeling applied to each kind of data. Findings show significant misalignment between these two data sources. Self-reported goals stated before learning do not translate into goal-relevant behaviors tracked using trace data collected during learning activities. While learners generally articulate an orientation towards mastery learning in self-report surveys, behavioral trace data showed a higher incidence of less engaged learning activities. These findings call into question the utility of survey-based measures when up-to-date achievement goal data are needed. Our results advance methodological and theoretical understandings of achievement goals in the modern age of learning analytics.