Dynamic occlusion handling for real-time AR applications

Abstract

Augmented reality (AR) allows computer generated graphics to be overlaid in images or video captured by a camera in real time. This technology is often used to enhance perception by providing extra information or simply by enriching the experience of the user. AR offers a significant potential in many applications such as industrial, medical, education and entertainment. However, for AR to achieve the maximum potential and become fully accepted, the real and virtual objects within the user's environment must become seamlessly integrated. Three main types of problems arise when we try to achieve this effect: Illumination issues, tracking difficulties and occlusion troubles. In this work we present an algorithm to handle AR occlusions in real time. Our approach uses raw depth information of the scene to realize a rough foreground/background segmentation.We use this information, as well as details from color data to estimate a blending coefficient and combine the virtual objects with the real objects into a single image. After experimenting with different scenes we show that our approach is able to produce consistent and aesthetically pleasing occlusions between virtual and real objects, with a low computational cost. Furthermore, we explore different alternatives to improving the quality of the final results while overcoming limitations of previous methods.