Evaluating GPT as automated analyzer for detecting students’ erroneous mental models in programming education

Abstract

Purpose: This study evaluates Large Language Models as automated analyzers for detecting students’ erroneous mental models in programming education, addressing the scalability barrier of labor-intensive manual analysis. Building upon preliminary GPT-4 findings, this investigation extends to GPT-5, assessing generational improvements in reliability, error patterns, and computational feasibility. Methods: GPT-4 and GPT-5 analyzed 1,500 student responses to 24 C++ programming questions, yielding 5,300 binary judgments for 27 hypothesized mental models compared against expert faculty evaluations. Inter-rater agreement was quantified through percentage agreement and Cohen’s Kappa, with confusion matrix analysis revealing systematic error patterns. Results: GPT-5 achieved 82.7% raw agreement with faculty (vs. GPT-4’s 66.9%) and Cohen’s Kappa of = 0.38 (vs. 0.18), though below the 0.60-−0.80 inter-human threshold. Both models exhibited false positive bias, with GPT-5’s reduced rate (11.6% vs. 23.1%) driving improvement. GPT-5 demonstrated 7.6-fold speed improvement and 34% cost reduction ($12.51 vs. $19.06). Conclusion: Current LLMs achieve production-ready status within hybrid human-AI frameworks. GPT-5’s fair-to-moderate reliability combined with cost-performance improvements ($50 per 100-student cohort) establishes feasibility at educational scales. However, reliability insufficient for autonomous deployment supports hybrid configurations wherein LLMs serve as first-pass analyzers subject to systematic faculty review.