The Human-Built Environment-Natural Environment Relation-An Immersive Multisensory Exploration with 'System of a Sound'

Abstract

The relationship between human activity, the built environment, such as homes, offices, and schools, and the surrounding natural environment hide interaction patterns that can be better understood. Intelligent user interfaces today use dashboards with tables and figures, missing a way where people can innately relate to this relationship. We explored emotion-oriented data sonification as an experimental way for people to engage with data. Our audio-visual and gestural intelligent user interface, "System of a Sound", takes real-time data streams centred on the location where it is installed and uses a large language model to choose music samples that elicit matching emotions. The samples are combined using an artificial intelligence music engine to compose a live soundscape that reveals interaction patterns between human activity, the built environment, and the surrounding natural environment. This exciting IUI Demo offers conference participants emotion-oriented data sonification for data exploration.