Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Abstract

The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.