TSF: Two-Stage Sequential Fusion for 3D Object Detection

Abstract

There have been significant advances in 3D object detection using LiDAR and camera fusion for autonomous driving. However, it is surprisingly difficult to effectively design fusion location and fusion strategies for point cloud-based 3D object detection networks. In this paper, we propose a novel two-stage sequential fusion (TSF) method. In the first stage of fusion, TSF generates the enhanced point cloud by combining the raw point cloud and semantic information of image instance segmentation. In the second stage, the proposals generated by LiDAR baseline is used to complete the No-Maximum Suppression (NMS) together with the 2D object detection results. Numerous experiments on the KITTI validation set show that our method outperforms state-of-the-art multimodal fusion-based methods on the three classes in 3D performance (Easy, Moderate, Hard): cars (89.94%, 82.76%, 76.04), pedestrians (70.74%, 63.47%, 56.56%), and cyclists (84.72%, 64.22%, 56.78%). In ablation, we analyze the augmented effect of fusion module on the LiDAR baseline detection capability, and study the best trade-off between running time and accuracy.