Revisiting Sparsity Invariant Convolution: A Network for Image Guided Depth Completion

Abstract

The limitation of LiDAR (Light Detection And Ranging) sensor causes the general sparsity of produced depth measurement. However, the sparse representation of the world is insufficient for applications such as 3D reconstruction. Thus, depth completion is an important computer vision task in which a synchronized RGB image is commonly available. In this paper, we propose a deep neural network to tackle this image guided depth completion problem. By revisiting the sparsity invariant convolution and revealing how it can be used in a novel approach, we propose three mask aware operations to process, downscale, and fuse sparse features. These operations explicitly consider the observation mask of its corresponding feature map. In addition, the structure of this network follows a novel scheme in which data from image and depth domain are processed by these proposed operations independently. Our proposed model achieves state-of-the-art performance on the KITTI depth completion benchmark. Furthermore, it presents a strong robustness for bearing input sparsity under different densities and patterns.