Following the Master's Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications

Abstract

Piano learning applications in Mixed Reality (MR) are a promising substitute for physical instruction when a piano teacher is absent. Existing piano learning applications that use visual indicators to highlight the notes to be played on the keyboard or employ video projections of a pianist provide minimal guidance on how the learner should execute hand movements to develop their technique in performance and prevent injuries. To address this gap, we developed an immersive first-person piano learning experience that uses a library of targeted visualizations of the teacher's hands and 3D traces of hand movements in MR. Seeing the piano teacher's hands while hearing the music is central to developing the novice's musical intuition. We introduced an end-to-end workflow to accurately capture the pianist's technical gestures and align them with the musical score. We recorded pianists playing technical exercises and music pieces. We developed a multimodal performance dataset (MPD) comprising virtual hand models, keyboard (MIDI) recordings and the corresponding music scores, and different visualizations of hand traces capturing movement. Finally, we developed Pianoverse, an MR application to assist piano learning, and performed exploratory user testing with novice piano players to understand the impact of multimodal representations of movement on skill learning. Our initial observations suggest that apprehending the movement traces of a recorded performance over a physical keyboard increases the learner's ability to position their body and hands correctly and to replicate hand gestures while playing from written music. Further research will focus on automating performance data collection and a comprehensive evaluation of the use of leading movement traces in piano learning.