The effects of compilation mechanisms and error message presentation on novice programmer behavior

Abstract

It is generally accepted that learning to program could be easier for many students. One of the most important components of this experience is the programming environment. Novices learn in a variety of environments, from basic command-line interfaces to industrystrength IDEs. These environments can differ substantially in compilation behavior and error message presentation arguably two of the most important mechanisms through which users interact with the programming language. In this study, we utilize Blackbox data to compare the programming behavior of thousands of users programming in Java, who all used BlueJ versions 3 and 4. These two versions differ drastically in terms of compilation behavior and error message presentation. BlueJ 3 is a click-to-compile editor that delivers text-based error messages from javac to the user, but only presents the first error message, even if the compiler produces several. BlueJ 4 automatically compiles in the background but retains click-to-compile ability. In addition, all error messages (not just the first) may be viewed by the user. We find that the programming experience and behavior of these users can be substantially affected by changes in these mechanisms, causing numbers of manual compilations, successful compilations, and error messages presented in each version to differ, in cases, markedly. Our results provide evidence on how changes in programming environment affect user behavior in conditions that reasonably control for variables other than the programming environment. This can inform the decisions of educators, tool designers, and HCI researchers in their work to make learning more effective for novice programmers.