Real-time image based lighting for 360-degree panoramic video

Abstract

This paper presents an effective approach to rendering virtual 3D objects using real-time image based lighting (IBL) with conventional 360 ◦ panoramic video. Raw 360 ◦ panoramic video captured in a low dynamic range setup is the only light source used for the realtime IBL rendering. Input video data is boosted to high dynamic range using inverse tone mapping. This converted video is then reconstructed into low-resolution diffuse radiance maps to speed up diffuse rendering. A mipmap-based specular sampling scheme provides fast GPU rendering even for glossy specular objects. Since our pipeline does not require any precomputation, it can support a live 360 ◦ panoramic video stream as the radiance map, and the process fits easily into a standard rasterization pipeline. The results provide sufficient performance for IBL via stereo head mounted display (HMD), an ideal device for immersive augmented reality films and games using 360 ◦ panoramic videos as both the lighting and backdrop for illumination composition.