Predicting student knowledge level from domain-independent function and content words

Abstract

We explored the possibility of predicting the quality of student answers (error-ridden, vague, partially-correct, and correct) to tutor questions by examining their linguistic patterns in 50 tutoring sessions with expert human tutors. As an alternative to existing computational linguistic methods that focus on domain-dependent content words (e.g., velocity, RAM, speed) in interpreting a student's response, we focused on function words (e.g., I, you, but) and domain-independent content words (e.g., think, because, guess). Proportional incidence of these word categories in over 6,000 student responses to tutor questions was automatically computed using Linguistic Inquiry and Word Count (LIWC), a computer program for analyzing text. Multiple regression analyses indicated that two parameter models consisting of pronouns (e.g., I, they, those) and discrepant terms (e.g., should, could, would) were effective in predicting the conceptual quality of student responses. Furthermore, the classification accuracy of discriminant functions derived from the domain-independent LIWC features competed with conventional domain-dependent assessment methods. We discuss the possibility of a composite assessment algorithm that focuses on both domain-dependent and domain-independent words for dialogue-based ITSs. © 2010 Springer-Verlag.