SS-SF: Piecewise 3D Scene Flow Estimation with Semantic Segmentation

Abstract

In order to address the issue of edge-blurring and improve the accuracy and robustness of scene flow estimation under motion occlusions, we in this article propose a piecewise 3D scene flow estimation approach with semantic segmentation, named SS-SF. First, we utilize the semantic optical flow to initialize the 3D plane and its rigid motion parameters, and then produce the initial mappings of pixel-to-segment and segment-to-plane of the input left and right image sequences. Second, we plan a novel energy function to optimize the initial mappings by using a semantic segmentation constraint term to regularize the classical scene flow model, which the optimized mappings are employed to update the assignment and motion parameters of each pixel. Third, we adopt the semantic label to extract the occlusion pixels and exploit an occlusion handling constraint to enhance the robustness of the scene flow estimation. Finally, we compare the proposed SS-SF model with several state-of-the-art approaches by using the KITTI and MPI-Sintel databases. The experimental results demonstrate that the proposed method has the advanced accuracy and robustness in scene flow estimation, especially owns the capacities of edge-preserving and occlusion handling.