Point Auto-Encoder and Its Application to 2D-3D Transformation

Abstract

PointNet has been shown to be an efficient way to encode the global geometric features of a point cloud representation of 3D objects based on supervised learning. Although it is quite interesting to have a decoder integrated into PointNet for possible unsupervised end-to-end learning, similar to conventional auto-encoders, only a few methods have been proposed to date. The proposed methods are shown to be able to reconstruct the given input point cloud from the corresponding global features through decoders, however, further improvements not only in their performance of training in terms of accuracy but also in their power of generalization in decoding testing samples. This paper presents a Point Auto-Encoder, or Point AE, which is implemented based on the novel semi-convolutional and semi-fully-connected layers proposed that can handle the problem of mapping from single global feature vector to massive numbers of 3D points. The proposed Point AE is not only simpler in its architecture but also more powerful in terms of training performance and generalization capability than state-of-the-art methods. The effectiveness of Point AE is well verified based on the ShapeNet and ModelNet40 dataset. Furthermore, in order to demonstrate the extended capability of Point AE, we apply Point AE to the automatic transformation of images from 2D to 3D and 3D to 2D.