Auraglyph: Handwritten Computer Music Composition and Design

Abstract

Effective software interaction design must consider all of the capabilities and limitations of the platform for which it is developed. To this end, we propose a new model for computer music system design on touchscreen devices, combining both pen/stylus input and multitouch gestures. Such a model surpasses the barrier of touchscreen-based keyboard input, preserving the primary interaction of touch and direct manipulation throughout the development of a complex musical program. We have implemented an iPad software application utilizing these principles, called ``Auraglyph." Auraglyph offers a number of fundamental audio processing and control operators, as well as facilities for structured input and output. All of these software objects are created, parameterized, and interconnected via stylus and touch input. Underlying this application is an advanced handwriting recognition framework, LipiTk, which can be trained to recognize both alphanumeric characters and arbitrary figures, shapes, and patterns.