Teaching introductory programming concepts through a gesture-based interface

Abstract

The goal of our research is to create and evaluate a visual and gesture-driven interface to teach computer programming to non-traditional programmers, typically school-age children. By making the interface more enjoyable for young students, we hope to keep students engaged and increase their attention span while learning how to program. Our system combines components from Google’s Blockly, a visual block programming language with drag-and-drop puzzle pieces, and Microsoft’s Xbox Kinect, which is used to perform skeletal tracking. We created pre-defined gestures to correspond to program functions and available actions, which were compiled from a survey conducted of over 100 grade-school students over three years who had very little to no programming experience before we met. After learning how to use Blockly and having a basic understanding of simple programming logic, the students were asked to create intuitive gestures for common programming constructs, while both standing up using full body movement, and sitting down at a desk, using only their hands. The specific programming constructs included in the survey were loops, conditionals, run program, and undo. To detect the gestures, we have implemented and evaluated a number of gesture matching algorithms. One challenge is that the size, shape, and path of the gestures varied considerably, so the data has to be normalized for any comparisons.