Effects of Automated Feedback in Scratch Programming Tutorials

Abstract

Block-based programming languages like Scratch are commonly used to introduce young learners to programming. While coding, learners may encounter problems, which may require teachers to intervene. However, teachers may be overwhelmed with help requests in a classroom setting, and in independent learning scenarios, teachers may not be available at all. Automated tutoring systems aim to help by providing hints, but misleading or confusing hints can be detrimental. To better understand the effects of automatically generated hints, in this paper we study a state-of-the-art hint generation system that provides suggestions when learners fail to complete a step in a programming tutorial. The system is evaluated using two cohorts of students aged 12-13, where one cohort receives only textual hints based on test failures while the other additionally receives visual next-step support in terms of illustrated code changes. We find that initially the automatically generated visual next-step hints increase the speed at which learners complete the steps of the tutorial and reduce the number of questions posed to teachers, without affecting the learners' overall understanding of their program negatively. However, with increasing complexity of the programs the quality of the hints degrades, thus calling for further research on improving hint generation systems.