Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations

Abstract

Deep learning appears as an appealing solution for automatic synthesizer programming (ASP), which aims to assist musicians and sound designers in programming sound synthesiz-ers. However, integrating software synthesizers into training pipelines is challenging due to their potential nondifferentiability. This work tackles this challenge by introducing a method to approximate arbitrary synthesizers. Specifically, a neural network is trained to map synthesizer presets onto an audio embedding space derived from a pretrained model. This facilitates the definition of a neural proxy that produces compact yet effective representations, thereby enabling the integration of audio embedding loss into neural-based ASP systems for black-box synthesizers. The authors evaluate the representations derived by various pretrained audio models in the context of neural-based methods for ASP and assess the effectiveness of several neural network architectures, including feedforward, recurrent, and transformer-based mod-els, in defining neural proxies. The proposed method is evaluated using both synthetic and handcrafted presets from three popular software synthesizers and assessed its performance in a synthesizer sound-matching downstream task. Although the benefits of the learned rep-resentation are nuanced by resource requirements, encouraging results were obtained for all synthesizers, paving the way for future research into the application of synthesizer proxies for neural-based ASP systems.