Interacting with Neural Radiance Fields in Immersive Virtual Reality

Abstract

Recent advancements in the neural radiance field (NeRF) technology, in particular its extension by instant neural graphics primitives, provide tremendous opportunities for the use of real-time immersive virtual reality (VR) applications. Moreover, the recent release of an immersive neural graphics primitives framework (immersive-ngp) brings real-time, stereoscopic NeRF rendering to the Unity game engine. However, the system and application research combining NeRF and human-computer interaction in VR is still at the very beginning. In this demo, we present multiple interactive system features for immersive-ngp with design principles focusing on improving the usability and interactivity of the framework for small to medium-scale NeRF scenes. We demonstrate that these new feature implementations such as exocentric manipulation, VR tunneling effects, and immersive scene appearance editing enable novel VR-NeRF experiences, for example, for customized experiences in inspecting a particle accelerator environment.