Situated, Perceptual, Emotive and Cognitive Music Systems: a psychologically grounded approach to interactive music composition.

Abstract

This thesis introduces a novel situated interactive composition system called the SMuSe (for Situated Music Server) that is grounded on principles of modern cognitive science, provides perceptual control of sound synthesis and includes emotional feedback. It il- lustrates both a new music composition paradigm and a synthetic psychology approach to the study of music perception, emotion and cognition. The SMuSe is composed of cognitively plausible modules implemented as a hierarchy of musical agents and relies on distributed control, parallelism, emergence and embodiment. By interacting with its en- vironment, which provides feedback via multiple sensors, the system generates complex adaptive affective musical structures. Focusing on the micro-level of sound generation, we present two complementary techniques that give high-level perceptual control over low-level sound synthesis parameters. In a first implicit approach, a support vector ma- chine algorithm learns to automatically map perceptual features such as loudness, pitch and brightness onto additive synthesis parameters. In a second approach, a physically- inspired synthesis model provides explicit access to perceptual and physical parameters such as pitch, loudness, brightness, attack time, inharmonicity and damping. Moving from the study of music generation and control towards the study of the musical ex- perience itself, we then evaluate how the music generated by the SMuSe influences the listeners’ emotional responses. A first psychoacoustics experiment shows the significant influence of structural (scale, register, harmony), expressive (velocity, tempo, articula- tion) and timbre (brightness, attack time, spectral flux) parameters on the emotional scales of valence, arousal and dominance. An additional large scale experiment involving dementia patients (an illness known to induce cognitive and affective deficits) shows that specific sound features (e.g. low loudness, low brightness) provoke specific emotional re- sponses within the patients (e.g. low stress). Moreover, the patients’ emotional responses differ from the age-matched control group, and the analysis shows an increased emotional sensitivity to sounds as the severity of the disease increases. These results suggest that sound-based therapy and diagnosis for dementia are possible. Finally, the maturity and flexibility of the SMuSe music system are demonstrated by a series of real-world applica- tions including the sonification of a mixed-reality space, a study on physiologically-based musical interaction, a neurofeedback musical interface, a closed loop system based on reinforcement learning of emotional feedback, and a large scale multimedia performance using brain-computer interfaces. A situated, perceptive, emotive and cognitive approach to the design of musical systems paves the way for new applications for therapy but also for interactive gaming and novel physiologically-based instruments. Our approach pro- vides a well-grounded paradigm to develop advanced synthetic aesthetics system that can inform our understanding of the psychological processes on which they rely.