High-Fidelity Depth Map Reconstruction System with RGB-Guided Super Resolution CNN and Cross-Calibrated Chaos LiDAR

Abstract

High-fidelity 3D models are essential for immersive virtual and augmented reality (VR/AR) applications. However, the performance of current 3D recording devices is limited in several scenarios, such as dim light environments, long-distance measurements, and large-scale objects. Therefore, their applicability to indoor scenes is hindered. In this work, we propose a depth map reconstruction system that integrates an RGB-guided depth map super-resolution convolutional neural network (CNN) into a stand-alone Chaos LiDAR depth sensor. This system provides highly accurate depth estimates in various scenarios, particularly for indoor scenes with dim lighting or long distances ranging from 4 m to 6 m. We address two design challenges to maximize the quality of the reconstructed depth map of the system. First, the misalignment across RGB-depth sensors is addressed using a two-stage calibration pipeline. Second, the lack of large-scale real-world LiDAR datasets is addressed by generating a large-scale synthetic dataset and adopting transfer learning. Experimental results show that our proposed system significantly outperforms the commercial RGB-D recording device RealSense D435i in terms of subjective visual perception, precision, and density of depth estimates, making it a promising solution for general indoor scene recording.