Extending the Visual Arts Experience: Sonifying Paintings with AI

Abstract

Sonification of visual information is a relatively new research line that aims to create a new way to access and experience visual displays, especially for the visually impaired. When applied to artworks, sonification needs to translate the aesthetic experience as well. This is attempted via a handful studies in the literature, where most of the transformation and music generation is done manually, or only by using the low level visual features of artworks. In this paper, we present a sonification model that uses both low level and high level features such as color, edge information, saliency, object and scene detection to create a pleasant and descriptive sonification of artworks with the use of a fully automatic pipeline. The results of the model are tested via interviews done with experts in music theory and generative music models. We found a high agreement among experts for the evaluation of a small set of sonified paintings. Addition of high level features such as sounds extracted from the scene played a big role in this. Among the challenges observed during the interviews was the need to add emotion and mood information as well as semantic information to the sonification in order to create more descriptive melodies and sounds. The complexity and ambiguity of the visual information generated the most disagreement among experts both in their interpretation of the paintings as well as their sonifications.