Spatio-temporal consistency in depth video enhancement

Abstract

In recent years, new active range sensors have been developed for 3D data acquirement, such as time-of-flight cameras. These sensors enable acquiring of range data at video rate and are suited for dynamic environment. Unfortunately, the resolution of the range data is quite limited and the captured data are typically contaminated by noise. In this paper, we propose a novel method for depth video enhancement. Using high resolution color video as guidance reference, we iteratively refine the input depth map based on a newly presented linear filter model, in terms of both its spatial resolution and depth precision. The linear filter has a good edge-preserving property and a runtime independent of filter size, which fulfills both accuracy and speed requirements. For temporally consistent estimate on depth video, we extend the method into temporally neighboring frames. Simple optical flow and patch-based similarity measure are used to obtain accurate depth in an efficient manner. Experimental results show that the proposed method greatly improves the quality and boosts the resolution of range data while achieving high computational efficiency. We also show that the temporally consistent constraint addresses a flickering problem and improves the accuracy of depth video. © 2013 by JSME.