Mixed reality merging of endoscopic images and 3-D surfaces

Abstract

In image-guided neurosurgery, “mixed reality” merging has been used to merge video images with an underlying computer model. We have developed methods to map intra-operative endoscopic video to 3D surfaces derived from pre-operative scans for enhanced visualization during surgery. We acquired CT images of a brain phantom, and digitized endoscopic video images from a tracked neuro-endoscope. Registration of the phantom and CT images was accomplished using markers that could be identified in both spaces. The endoscopic images were corrected for radial lens distortion, and mapped onto surfaces extracted from the CT images via a ray-traced texture-mapping algorithm. The localization accuracy of the endoscope tip was within 1.0 mm. The mapping operation allows the endoscopic images to be permanently painted onto the surfaces. Our method allows panoramic and stereoscopic visualization from arbitrary perspectives (though the original endoscopic video was monoscopic) and navigation of the painted surface after the procedure.