Object spatial localization by fusing 3D point clouds and instance segmentation

Abstract

Real-time detection and acquisition of localization information of instance targets in real three-dimensional space plays an important role in application scenarios such as virtual reality simulation and digital twinning. The existing spatial localization methods without the aid of lidar and other equipment often have problems in restoring the real scale. In order to overcome this problem and achieve more accurate object spatial localization, an object spatial localization by fusing 3D point clouds and instance segmentation is proposed. This method obtains sparse 3D point cloud data by binocular stereo matching, which is used to describe the real scale and spatial location information of the object. Then uses deep learning method to perform monocular instance segmentation on the specific category target of interest, and the segmentation result is used as the front/background prior information to complete the coordinate correction and densification of the 3D point cloud data inside and outside the object contour. Compared with the unsupervised depth estimation methods based on deep learning, our method can quickly and accurately achieve the three-dimensional precise localization of the instance target and its various components in real-world scenes, and the accuracy in the indoor scene is more than 90%.