Toward non-intrusive assessment in dialogue-based intelligent tutoring systems

Abstract

This paper describes a study whose goal was to assess students’ prior knowledge level with respect to a target domain based solely on characteristics of the natural language interaction between students and a state-of-the-art conversational ITS. We report results on data collected from two conversational ITSs: a micro-adaptive-only ITS and a fully adaptive (micro- and macro-adaptive) ITS. Our models rely on both dialogue and session interaction features including time-on-task, student-generated content features (e.g., vocabulary size or domain-specific concept use), and pedagogy-related features (e.g., level of scaffolding measured as number of hints). Linear regression models were explored based on these features in order to predict students’ knowledge level, as measured with a multiple-choice pre-test, and yielded in the best cases an r = 0.949 and adjusted r-square = 0.878.