RealPoint3D: An Efficient Generation Network for 3D Object Reconstruction from a Single Image

Abstract

The generation of 3D models from a single image has recently received much attention, based on which point cloud generation methods have been developed. However, most current 3D reconstruction methods only work for relatively pure backgrounds, which limit their applications on real images. Meanwhile, more fine-grained details are required to provide finer models. This paper proposes an end-to-end efficient generation network, which is composed of an encoder, a 2D-3D fusion module, and a decoder. First, a single-object image and a nearest-shape retrieval from ShapeNet are fed into the network; then, the two encoders are integrated adaptively according to their information integrity, followed by the decoder to obtain fine-grained point clouds. The point cloud from the nearest shape effectively instructs the generation of finer point clouds. To have a consistent spatial distribution from multi-view observations, our algorithm adopts projection loss as an additional supervisor. The experiments on complex and pure background images show that our method attains state-of-the-art accuracy compared with volumetric and point set generation methods, particularly toward fine-grained details, and it works well for both complex backgrounds and multiple view angles.