Mobile Volumetric Video Streaming System through Implicit Neural Representation

Abstract

Volumetric video (VV) emerges as a new video paradigm with six degree-of-freedom (DoF) immersive viewing experience. Most existing VV systems focus on the point cloud (PtCl)-based architecture, which is however far from effective due to the huge video size, unrealistic color variations, and specialized player platform requirement. The recent advance of implicit neural representations (INR) such as NeRF brings great opportunities to VV given its potential in creating photorealistic 3D appearances and lighting consistency. However, there still exist arduous challenges in many aspects such as model training, display rendering, streaming optimization, and system implementation. To address the above challenges, we develop NeRVo, an INR-based VV representation for mobile VV. NeRVo improves the training and rendering speed over 300x and 1000x with photorealism, mobile compatibility, and desirable datarates compared to NeRF. We adopt NeRVo as a building block, design and implement a holistic INR-enhanced VV streaming system VoINR.