Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT

Abstract

Identifying the misconceptions of novice programmers is pertinent for informing instructors of the challenges faced by their students in learning computer programming. In the current literature, custom tools, test scripts were developed and, in most cases, manual effort to go through the individual codes were required to identify and categorize the errors latent within the students’ code submissions. This entails investment of substantial effort and time from the instructors. In this study, we thus propose the use of ChatGPT in identifying and categorizing the errors. Using prompts that were seeded only with the student’s code and the model code solution for questions from two lab tests, we were able to leverage on ChatGPT’s natural language processing and knowledge representation capabilities to automatically collate frequencies of occurrence of the errors by error types. We then clustered the generated error descriptions for further insights into the misconceptions of the students. The results showed that although ChatGPT was not able to identify the errors perfectly, the achieved accuracy of 93.3% is sufficiently high for instructors to have an aggregated picture of the common errors of their students. To conclude, we have proposed a method for instructors to automatically collate the errors latent within the students’ code submissions using ChatGPT. Notably, with the novel use of generated error descriptions, the instructors were able to have a more granular view of the misconceptions of their students, without the onerous effort of manually going through the students’ codes.