Bowing detection for erhu learners using yolo deep learning techniques

Abstract

The traditional ways to learn musical instruments are having lessons with teachers, or self-paced by teaching videos and books. Nowadays, the musical instrument self-learning systems are in the ascendant. However, it is difficult for a musical instrument learner to know if his playing posture is correct unless the teacher provides guidance in person and corrects it immediately. Due to the breakthrough development of computer vision technology, computers with camera lenses can act as teachers’ eyes to detect the learners’ movements and give some assistance for their self-learning. Erhu, a kind of Chinese traditional stringed instrument, is getting popular worldwide. In this research, taking the erhu bowing method as an example, we use YOLO object detection technology to track the trajectory of the bow in the video and evaluate whether it meets the level and straightness requirements. For the purpose, we first define the measurement methods of level and straightness and design a system to record the progress. In 2016, Joseph Redmon et al. proposed a real-time object detection method called YOLO, “You Only Look Once”. The multi-layer CNN architecture is executed only once, which greatly improves the detection speed. To track the movement of erhu’s bow, we must at least detect the coordinates of two objects, the erhu bow and the right hand of the player who holding the bow. Our images data set to train the object detection model came from several Chinese music society in Taiwan. The experimental results show that the model can successfully mark the desired objects. Finally, the study found that even senior erhu players still have some deviations, but they are more stable than beginners. Self-correction through system prompts would help learners to reduce the bias problem.