Spatial Audio in 360° Videos: Does it influence Visual Attention?

Abstract

Immersive technologies are rapidly gaining traction across a variety of application domains. 360° video is one such technology, which can be captured with an omnidirectional multi-camera arrangement. With a Virtual Reality (VR) Head Mounted Display (HMD), users have the freedom to look in any direction they wish within the scene. While there is a plethora of work focused on modeling visual attention (VA) in VR, little research has considered the influence of the audio modality on VA in VR. It is well known that audio has an important role in VR experiences. Listeners can experience sound in all directions with high quality spatial audio. One such technique, Ambisonics or 3D audio, provides a full 360° sound soundscape. In this paper, the results of an empirical study that examined how (if at all) spatial audio influences visual attention in 360° videos are presented. The same videos were accompanied with either non-spatial (stereo) or spatial (third order Ambisonics) sound. Pose and gaze fixations, pupil diameter, audio energy maps (AEMs) and associated analysis for 20 users watching ten 360° videos across various categories (in Indoor and Outdoor conditions) are presented. The findings reveal that users have different viewing patterns and physiological responses for the different sound conditions. With the increasing use of spatial audio in VR and 360° videos, this knowledge can help develop effective techniques for optimizations in terms of processing, encoding, distributing, and rendering content.