Can computers compare student code solutions as well as teachers?

Abstract

In introductory programming courses it is common to demand from students exercises based on the production of code. However, it is difficult for the teacher to give fast feedback to the students about the main solutions tried, the main errors and the drawbacks and advantages of certain solutions. If we could use automatic code comparison algorithms to build visualisation tools to support the teacher in analysing how each solution provided is similar or different from another, such information would be able to be rapidly obtained. However, can computers compare students code solutions as well as teachers? In this work we present an experiment in which we have requested teachers to compare different code solutions to the same problem. Then we have evaluated the level of agreement among each teacher comparison strategy and some algorithms generally used for plagiarism detection and automatic grading. We found out a maximum rate of 77% of agreement between one of the teachers and the algorithms, but a minimum agreement of 75%. However, for most of the teachers, the maximum agreement rate was over 90% for at least one of the automatic strategies to compare code. We have also detected that the level of agreement among teachers regarding their personal strategies to compare students solutions was between 62% and 95%, which shows that there may be more agreement between a teacher and an algorithm than between a teacher and one of her colleagues regarding their strategies to compare students' solutions. The results also seem to support that comparison of students' codes has significant potential to be automated to help teachers in their work. Copyright © 2014 ACM.