SALI360: Design and implementation of saliency based video compression for 360° video streaming

Abstract

In accordance with the recent enhancement of display technology, users demand a higher quality of streaming service, which escalates the bandwidth requirement. Considering the recent advent of high FPS (frame per second) 4K and 8K resolution 360° videos, such bandwidth concern further intensifies in 360° Virtual Reality (VR) content streaming even at a larger scale. However, the currently available bandwidth in most of the developed countries can hardly support the bandwidth required to stream such a scale of content. To address the mismatch between the demand on higher quality of streaming service and the saturated network improvement, we propose SALI360 that practically solves the mismatch by utilizing the characteristics of the human vision system (HVS). By pre-rendering a set of regions - where viewers are expected to fixate - on 360° VR content in higher quality than the other regions, SALI360 improves viewers' quality of perception (QoP) while reducing content size with geometry-based 360° content encoding. In our user experiment, we compare the performance of SALI360 to the existing 360° content-encoding techniques based on 20 viewers' head movement and eye gaze traces. To evaluate viewers' QoP, we propose FoL (field of look) that captures viewers' quality perception area in the visual focal field (8°) rather than a wide (around 90°) field of view (FoV). Results of our experimental 360° VR video streaming show that SALI360 achieves 53.3% of PSNR improvement in FoL, while gaining 9.3% of PSNR improvement in FoV. In addition, our subjective study on 93 participants verifies that SALI360 improves viewers' QoP in the 360° VR streaming service.