A virtual musical instrument for 3D performance with short gestures: Exploring mapping strategies with virtual reality

Abstract

We present a new digital virtual instrument designed, built and played using 3D interactive technologies, with interest in reducing both learning time and difficulty in playing musical phrases and chords across the musical scale. Design concepts stress on a smart note mapping, rely on bi-manual interaction, and intend an optimized gestural performance on a virtual keyboard to shorten playing trajectories. The proposed keyboard is built stacking note keys in multiple lines, and the note mapping allows performing typical musical intervals and patterns with short gestures. This implementation employed an Oculus Rift head-mounted display and a Razer Hydra for gesture input. The combination of 3D visualization, natural interaction and the proposed mapping is meant to contribute to ease musical performance by allowing fast execution of specific note sequences. The instrument concept encapsulates complexity by creating a fluid way for musicians to perform gesture patterns that would otherwise require non-trivial motor skills.