RoboPianist - A Computer Vision Based Artificial Intelligence Application for Hand Gesture Recognition

Abstract

In this paper, we present RoboPianist, a new artificial intelligence and robotics-based edutainment application for solfège learning. This project aims to create an interactive gesture recognition system that can recognize gestures that are expressed similarly the way how solfège is learned in music classes. For this, our setup consists of a small robotic arm that is connected to a computer to press and release the keys on an electric piano. The computer sends the controlling signals to the robot based on an artificial intelligence-based computer vision pipeline. The potential elements in the different stages of the pipeline were compared to each other in terms of accuracy, efficiency and model parameters, and were chosen accordingly. Our pipeline consists of models for three main functionalities: hand keypoint detection (MediaPipe); static hand signal recognition (a custom MLP model); and dynamic gesture recognition (a GRU model with semantic fusion). This way, our application can recognize the seven hand signals that are based on Kodály’s work, and with the forth and back movement understand the intent of key-press and release of a music note. The system presented high accuracy in the recognition of the hand signals and also the gestures, thus enabling fluent usage in music education.