High-Quality 3D Reconstruction with Depth Super-Resolution and Completion

Abstract

The 3D reconstruction is an important topic in computer vision with many applications, such as robotics and augmented reality. Since the raw depth images captured by consumer RGB-D cameras are often low resolution (LR), noisy, and incomplete. How to obtain high-quality 3D models with a consumer RGB-D camera is still a challenge for the existing systems. In this paper, we propose a new depth super-resolution and completion method implemented in a deep learning framework and build a high-quality 3D reconstruction system. We first improve the resolution of LR depth image with a depth super-resolution network and remove the outliers in high-resolution (HR) depth image based on gradient saliency. To further enhance the quality of HR depth image with the guide of HR color image, we learn surface normal and occlusion boundary images from the corresponding HR color image through two deep fully convolutional networks. In particular, the blurriness of HR color image is also detected and pixel-wise quantized. Finally, we obtain a completed HR depth image by optimizing the HR depth image with the surface normal, occlusion boundary, and color image blurriness. We have carried out qualitative and quantitative evaluations with baseline methods on public datasets. The experimental results demonstrate that our method has better performance both on single depth image enhancement and 3D reconstruction.