Simplifying indoor scenes for real-time manipulation on mobile devices

Abstract

Having precise measurements of an indoor scene is important for several applications - e.g.augmented reality furniture placement - whereas geometric details are only needed up to a certain scale. Depth sensors provide a highly detailed reconstruction but mobile phones are not able to display and manipulate these models in real-time due to the massive amount of data and the lack of computational power. This paper therefore aims to close this gap and provides a simplification of indoor scenes. RGB-D input sequences are exploited to extract wall segments and object candidates. For each input frame, walls, ground plane and ceiling are estimated by plane segments, object candidates are detected using a state-of-the-art object detector. The objects’ correct poses and semantic types are gathered by exploiting a 3D CAD dataset and by introducing a Markov Random Field over time. A vast variety of experiments outline the practicability and low memory consumption of the resulting models on mobile phones and demonstrate the ability of preserving precise 3D measurements based on a variety of real indoor scenes.