A protocol for simulated experimentation of automated grading systems

Abstract

Educational systems providing Automated Grading can be very useful for both learner and teacher. After their design and implementation, systems providing automated grading have to be tested and validated, which let the need for real world experiments arise. In this paper we describe an approach to “simulated experiments” that we hope can ease the task of the researchers developing the above mentioned systems. The concept of our proposal is in defining a simulated class by means of statistical distributions of the features that model the student and her/his work in the system. Since such features can be very different from a system to another, we try and keep the description of our framework as general as possible, and define a simulated class by the distribution of the grades that the learners should get (and the system should infer, if correctly working). We show the use of our approach with two systems, that do grading and model the students quite differently. We conclude that our simulation framework can be beneficial in validating an automated grading system, and in allowing to reflect on possible updates of the system, to make its grading more correct.