Tissue surface reconstruction aided by local normal information using a self-calibrated endoscopic structured light system

Abstract

The tissue surface shape provides important information for both tissue pathology detection and augmented reality. Previously a miniaturised structured light (SL) illumination probe (1.9 mm diameter) has been developed to generate sparsely reconstructed tissue surfaces in minimally invasive surgery (MIS). The probe is inserted through the biopsy channel of a standard endoscope and projects a pattern of spots with unique spectra onto the target tissue. The tissue surface can be recovered by light pattern decoding and using parallax. This paper introduces further algorithmic developments and analytical work to allow free-hand manipulation of the SL probe, to improve the light pattern decoding result and to increase the reconstruction accuracy. Firstly the “normalized cut” algorithm was applied to segment the light pattern. Then an iterative procedure was investigated to update both the pattern decoding and the relative position between the camera and the probe simultaneously. Based on planar homography computation, the orientations of local areas where the spots are located in 3D space were estimated. The acquired surface normal information was incorporated with the sparse spot correspondences to constrain the fitting of a thin-plate spline during surface reconstruction. This SL system was tested in phantom, ex vivo, and in vivo experiments, and the potential of applying this system in surgical environments was demonstrated.