Augmented Objects as Portals into Virtual Worlds:Using Audio to Create Immersive Experiences in Extended Realities

Abstract

Building on a long history of the integration of new technology into the arts, in this research we set out to develop and evaluate new methods for creating immersive, audio-centric artworks in a Mixed Reality (MR) context. We employ multiple levels of spatial audio, both within the room as well as the umbrellas that participants navigate the space with, in order to provide a 6 degree of freedom immersive experience. Furthermore, we use dynamic projection mapping, again both throughout the global environment, and on the umbrellas, to create immersive experiences. This allows the umbrellas to function as so-called Augmented Objects that facilitate the composition and navigation of Responsive Sonic Environments, thus encouraging participants to examine their own perceptions of reality while incorporating them into the larger artistic installation. Additionally, we explore the creation of Spatial Metacompositions, navigable environments that function as performance spaces for the participants who interact with them. Finally, we present the technical foundations necessary to create such systems and discuss techniques to effectively employ them in one's artistic practice.