SketchSynth: Cross-Modal Control of Sound Synthesis

Abstract

This paper introduces a prototype of SketchSynth, a system that enables users to graphically control synthesis using sketches of cross-modal associations between sound and shape. The development is motivated by finding alternatives to technical synthesiser controls to enable a more intuitive realisation of sound ideas. There is strong evidence that humans share cross-modal associations between sound and shapes, and recent studies found similar patterns when humans represent sound graphically. Compared to similar cross-modal mapping architectures, this prototype uses a deep classifier that predicts the character of a sound rather than a specific sound. The prediction is then mapped onto a semantically annotated FM synthesiser dataset. This approach allows for a perceptual evaluation of the mapping model and gives the possibility to be combined with various sound datasets. Two models based on architectures commonly used for sketch recognition were compared, convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In an evaluation study, 62 participants created sketches from prompts and rated the predicted audio output. Both models were able to infer sound characteristics on which they were trained with over 84% accuracy. Participant ratings were significantly higher than the baseline for some prompts, but revealed a potential weak point in the mapping between classifier output and FM synthesiser. The prototype provides the basis for further development that, in the next step, aims to make SketchSynth available online to be explored outside of a study environment.