Hybrid Mobile Vision for Emerging Applications

Abstract

While mobile applications have greatly benefited from 2D computer vision algorithms such as object detection and classification, there is limited research on exploring 3D vision that is enabled by the increasing availability of depth cameras and LiDAR scanners on mobile devices. In this paper, we propose a hybrid mobile vision system that intelligently combines 2D and 3D vision for improving the performance of emerging applications such as augmented and mixed reality and volumetric content analytics. Our research is motivated by and explores the key observation of the crucial latency-accuracy tradeoff between 2D and 3D vision. We present a research agenda with two principles for enhancing mobile vision stack, complementing 3D vision with its 2D counterpart by leveraging their diverse resource/accuracy profiles and processing 3D data (e.g., point clouds) with 2D vision cues for mitigating the high computation and storage costs.