Automatic Annotations by Augmented Reality-Enabled Laparoscopic Surgery

Abstract

Accurate labels of surgical procedures such as image segmentations or interaction labels are paramount for many of today's medical image computing tasks. Creating a dataset with these labels requires a great deal of manual work and relies on the involvement of medical experts, which is very time-consuming and costly. We propose a pathway for the automatic generation of such labels utilizing the spatial and temporal registration between a patient, the anatomical model, tracked surgical instruments, and the surgeon's view of the patient. These requirements for the automatic generation of labels are identical to the requirements of many navigated and augmented reality (AR) enabled surgeries. The AR system, through 3D registration, has the defining ability to accurately overlay real objects with their virtual counterparts. Our approach collects the complete raw data (e.g. video, tracking data, calibrations etc.) that feeds a live laparoscopic AR system for later analysis. By converting these complete recordings of the surgery into different representations, the AR system generates valuable datasets as mere by-products. Additionally, as our approach does not rely on visual input alone but on additional 3D information, the system can create labels even if the visual input is occluded or a tool interacts with tissue outside of the view of the laparoscopic camera. In this paper, we present a realization of this concept, then evolve this foundational idea into an interactive system that assists users in annotating surgical data. Finally, we gather and analyse feedback from six participants to evaluate the efficacy and user-friendliness of our system.