Impact of Contextual Tips for Auto-Gradable Programming Exercises in MOOCs

Abstract

Learners in Massive Open Online Courses offering practical programming exercises face additional challenges next to the actual course content. Beginners have to find approaches to deal with misconceptions and often struggle with the correct syntax while solving the exercises. The paper at hand presents insights from offering contextual tips in a web-based development environment used for practical programming exercises. We measured the effects of our approach in a Python course with 6,000 active students in a hidden A/B test and additionally used qualitative surveys. While a majority of learners valued the assistance, we were unable to show a direct impact on completion rates or average scores. We however noticed that users requesting tips took significantly longer and made more use of other assistance features of the platform than users in our control group. Insights from our study can be used to target beginners with more specific hints and provide additional, context-specific clues as part of the learning material.