Shape recovery of endoscopic videos by shape from shading using mesh regularization

Abstract

Endoscopic videos have been widely used for stomach diagnoses. It is of particular importance to obtain the 3D shapes, which enables observations from different perspectives so as to facilitate comprehensive and accurate diagnoses. However, obtaining 3D shapes is challenging for traditional multi-view 3D reconstruction methods, due to strong motion blurs, reflections, low spatial resolutions, non-rigid surfaces, and limited view angle shifts. In this work, we propose a mesh regularization for shape recovery based on cues derived from Shape-from-Shading (SfS). We recover shapes for all frames to generate a 3D video. In particular, a 3D mesh is optimized for every frame according to the 3D raw data obtained from SfS. Although the raw data contains errors and temporal jitters, our spatially and temporally optimized meshes can well approximate the underlying non-rigid surfaces, rendering temporally-stabilized meshes for 3D video display. Our experiments demonstrate the effectiveness of our method on many challenging endoscopic videos.