Automatic Evaluation of Programming Tasks Supported by Language Models

Abstract

Large language models are also increasingly used in education, both by students and teachers. Newly introduced LLM-based tools, such as Codex, Code Llama, and Microsoft's Copilot, show that LLMs are effective in code generation. In this paper, we evaluate the effectiveness of using LLMs in code evaluation. In programming courses, traditional manual code evaluation is very slow, and the participation of multiple instructors reduces consistency. Test-based code evaluation does not help novice programmers; for low-achieving students, the score attributed by such automatic tests is usually lower than what would have been gained after manual correction. Using LLMs might combine the benefits of manual correction and automatic testing. In order to apply LLMs to the task, detailed scoring guidelines were provided for the language models. The experiment involved 40 students in the first exam and 45 in the second exam in a C++ language course, with six teachers manually assessing the code fragments. The exams consisted of several different programming exercises, and the number of code snippets evaluated exceeded 350. We found a strong correlation between manual scoring and the scores given by the language models. Depending on the language model and the exam set examined, we measured correlation values between 0.6 and 0.9. The results of the experiment were significantly influenced by inconsistent teacher evaluations, so during the research phase, a single teacher was assigned to manual evaluation, which significantly improved the results. Based on these results, we repeated the experiment with an increased number of students (160) learning the Java programming language. The results again show a strong correlation (between 0.6 and 0.7). To clarify the results, we compared the models based on rubric specificity. Still, more research and testing are needed to explore the differences between manual evaluation and evaluation by large language models.