Reducing Sensing Errors in a Mixed Reality Musical Instrument

Abstract

This paper describes the design and evaluation of Netz, a novel mixed reality musical instrument that leverages artificial intelligence for reducing errors in gesture interpretation by the system. We followed a participatory design approach over three months through regular sessions with a professional musician. We explain our design process and discuss technological sensing errors in mixed reality devices, which emerged during the design sessions. We investigate the use of interactive machine learning techniques to mitigate such errors. Results from statistical analyses indicate that a deep learning model based on interactive machine learning can significantly reduce the number of technological errors in a set of musical performance tasks with the mixed reality musical instrument. Based on our findings, we argue that the application of interactive machine learning techniques can be beneficial for embodied, hand-controlled musical instruments in the mixed reality domain.