How Item and Learner Characteristics Matter in Intelligent Tutoring Systems Data

Abstract

AutoTutor-ARC (adult reading comprehension) is an intelligent tutoring system that uses conversational agents to help adult learners improve their comprehension skills. However, in such a system, not all lessons and items optimally serve the same purposes. In this paper, we describe a method for classifying items that are instructive, evaluative, motivational, versus potentially flawed based on analyses of items’ psychometric properties. Further, there is no a priori way of determining which lessons are optimal given the learner’s reading profile needs. To address this, we evaluate how assessing learner component reading skills can inform various aspects of learner needs on AutoTutor lessons. More specifically, we compare learners who were classified as proficient, underengaged, conscientious, versus struggling readers based on their experiences with AutoTutor. Together, these analyses suggest the utility of integrating assessments with instruction: efficient, adaptive learning at the lesson level, more efficient and valid post-testing, and consequently, recommendations for more targeted, adaptive pathways through the instructional program/system.