Autograding interactive computer graphics applications

Abstract

We present a system for the automated testing and grading of computer graphics applications. Our system runs, provides input to, and captures image and video output from graphical programming assignments. Instructors use a simple set of commands to script automated keyboard and mouse interactions with student programs at fixed times during execution. The resultant output - including plaintext standard output and mid-execution screenshots and GIFs - are displayed to the student to aid in debugging and ensure compliance with assignment specifications. Student output is automatically evaluated by basic text and image difference operations, or via an instructor-written validation method. We evaluate the success, implementation, and robustness of our design through deployment of this work in our university's senior undergraduate/graduate computer graphics course. In this course, students implement a variety of graphical assignments using OpenGL in C++. We summarize student feedback about the system gathered from anonymous end-of-term course evaluations. We provide anecdotal and quantitative evidence that the system improves student experience and learning by clarifying instructor expectations, building student confidence, and improving the consistency and efficiency of manual grading. This research has been implemented as an extension to Submitty, an open source, language-agnostic course management platform which allows automated testing and automated grading of student programming assignments. Submitty supports all levels of courses, from introductory to advanced special topics, and includes features for manual grading by TAs, version control, team submission, discussion forums, and plagiarism detection.