Light field reconstruction with a hybrid sparse regularization-pseudo 4DCNN framework

Abstract

The densely-sampled light field (LF) is highly desirable in many applications, such as 3D reconstruction, post-capture refocusing and virtual/augmented reality. However, it is costly and challenging to capture them because of the high dimensional nature. Existing view synthesis methods employ depth information for the densely-sampled LF reconstruction from the undersampled LF with a large disparity range, but fail to achieve non-Lambertian performance. In this article, a novel coarse-to-fine LF reconstruction method is proposed. We will develop a hybrid reconstruction framework that fuses model-based sparse regularization with deep learning. Specifically, sparse regularization with directional filter bank is utilized to solve the large baseline problem and gives a coarse densely-sampled LF. In addition, for those that cannot be recovered by classical model-based methods due to limited angular resolution, is estimated by learning a pseudo 4D convolutional neural network, and thereby offering a refinement on the intermediate LF. Extensive experiments demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.