Comparative Analysis of Learnersourced Human-Graded and AI-Generated Responses for Autograding Online Tutor Lessons

Abstract

Machine learning and artificial intelligence (AI) are ubiquitous, although accessibility and application are often misunderstood and obscure. Automatic short answer grading (ASAG), leveraging natural language processing (NLP) and machine learning, has received notable attention as a method of providing instantaneous, corrective feedback to learners without the time and energy demands of human graders. However, ASAG systems are only as valid as the reference answers, or training sets, they are compared against. We introduce an AI-based, machine learning method of autograding online tutor lessons that is easily accessible and user friendly. We present two methods of training set creation using: a subset of learnersourced, human-graded tutor responses from the lessons; and a surrogate model using the recently released AI-chatbot, ChatGPT. Findings indicate human-created training sets perform considerably better than AI-generated training sets (F1 = 0.84 and 0.67, respectively). Our straightforward approach, although not accurate enough for wide use, demonstrates application of directly available machine learning based NLP methods and highlights a constructive use of ChatGPT for pedagogical purposes that is not without limitations.