Multimodal Assessment of Network Music Performance

Abstract

The most common method of assessing the Quality of Musician's Experience (QoME) in Network Music Performance (NMP) is to perform a subjective study, where the participants evaluate their experience via questionnaires. Translating experiences into metrics is not an exact science, though: in our recent study on the effects of audio delay and quality in the QoME of NMP, the responses had a high variance and were inconsistent. To strengthen our confidence in the results of the subjective study, we analyzed video recordings of the participants using machine learning. Specifically, we used Facial Expression Recognition (FER) to detect the emotions felt by the participants and then compare them with their questionnaire responses. In addition to pointing out interesting phenomena that were not apparent from the questionnaires, this multimodal analysis showed analogies between the emotions felt (as captured by FER) and the emotions expressed (as captured by the responses).