Fooling it - student attacks on automatic short answer grading

Abstract

Modern machine learning approaches have been shown to be vulnerable to adversarial attacks in many fields. This is a critical weakness, especially for models that are expected to function in an adversarial environment, such as automatic grading models in exams. However, as most of these attacks are either limited in their success rate, their applicability in diverse scenarios or require mathematical expertise of the attacker, the question arises to which extent students themselves are even capable of fooling state-of-the-art grading models. This work aims to investigate this question for the short answer question format. For this purpose, we tasked students of an educational technologies university course with probing the state-of-the-art automatic short answer grading model for weaknesses. Of the fourteen active participants, only one reported the model to be sufficiently free of deficits. The following weaknesses were identified by the students: a disregard for negation, no plagiarism detection, correct answers not being predicted as such and oversensitivity to small linguistic changes in answers, triggers, and keywords.