PianoHandSync: An Alignment-based Hand Pose Discrepancy Visualization System for Piano Learning

Abstract

Video-based lessons are becoming a popular way for distance piano education. However, limited by the fixed camera angle, a video is difficult to tell precise 3D hand posture, which is one of the most essential factors for learning piano. This paper presents a visualization system providing the intuitive discrepancy of hand postures in two piano performance videos. Through a motion capture system, the estimated 3D postures are visualized and discrepancies based on distinct metrics are displayed, integrated with modular functions assisting skill acquisition. A pilot study proves that the proposed visualization can be a supplementary means for only video-based lessons in terms of correcting hand postures and fingering.