LS3D: Single-View Gestalt 3D Surface Reconstruction from Manhattan Line Segments

Abstract

Recent deep learning algorithms for single-view 3D reconstruction recover rough 3D layout but fail to capture the crisp linear structures that grace our urban landscape. Here we show that for the particular problem of 3D Manhattan building reconstruction, the explicit application of linear perspective and Manhattan constraints within a classical constructive perceptual organization framework allows accurate and meaningful reconstructions to be computed. The proposed Line-Segment-to-3D (LS3D) algorithm computes a hierarchical representation through repeated application of the Gestalt principle of proximity. Edges are first organized into line segments, and the subset that conforms to a Manhattan frame is extracted. Optimal bipartite grouping of orthogonal line segments by proximity minimizes the total gap and generates a set of Manhattan spanning trees, each of which is then lifted to 3D. For each 3D Manhattan tree we identify the complete set of 3D 3-junctions and 3-paths, and show that each defines a unique minimal spanning cuboid. The cuboids generated by each Manhattan tree together define a solid model and the visible surface for that tree. The relative depths of these solid models are determined by an L1 minimization that is again rooted in a principle of proximity in both depth and image dimensions. The method has relatively fewer parameters and requires no training. For quantitative evaluation, we introduce a new 3D Manhattan building dataset (3DBM). We find that the proposed LS3D method generates 3D reconstructions that are both qualitatively and quantitatively superior to reconstructions produced by state-of-the-art deep learning approaches.