Exploring Error State in "time-on-Task" Calculations at Scale

Abstract

Time-on-task has been shown to predict student performance in Computer Science courses [3], making it a useful tool for teachers to identify which students need extra support. Previous work has found that fine-grained metrics (e.g. keystroke-derived) for time-on-task produce stronger predictions of performance when compared to coarse-grained metrics (e.g. submission-based). This poster starts by replicating previous findings at scale, specifically that keystroke-derived time-on-task metrics predict grades at the assignment level. We attempt to add more granularity to our time-on-task metric by leveraging the exit code of the previous compile/run attempt to subdivide student time-on-task into time in an error state and time in an error-free state. We compare the predictive power of (1) keystroke-derived time-on-task, (2) error-free time-on-task, (3) error time-on-task, and (4) the ratio between error-free and error state time-on-task against assignment grades.