Marker-less AR in the hybrid room using equipment detection for camera relocalization

Abstract

Augmented reality (AR) permits clinicians to visualize directly in their field of view key information related to the performance of a surgery. To track the user’s viewpoint, current systems often use markers or register a reconstructed mesh with an a priori model of the scene. This only allows for a limited set of viewpoints and positions near the patient. Indeed, markers can be intrusive and interfere with the procedure. Furthermore, changes in the positions of equipment or clinicians can invalidate a priori models. Instead, we propose a marker-free mobile AR system based on a KinectFusion-like approach for camera tracking and equipment detection for camera relocalization. Our approach relies on the use of multiple RGBD cameras: one camera is rigidly attached to a hand-held screen where the AR visualization is displayed, while two others are rigidly fixed to the ceiling. The inclusion of two static cameras enables us to dynamically recompute the 3D model of the room, as required for relocalization when changes occur in the scene. Fast relocalization can be performed by looking at an equipment that is not required to remain static. This is particularly of advantage during hybrid surgeries, where an obvious choice for such an equipment is the intraoperative imaging device, which is large, can be seen in all views, but can also move. We propose to detect the equipment using a template based approach and further make use of the static cameras to speed-up the detection in the moving view by dynamically adapting the subset of tested templates according to the actual room layout. The approach is illustrated in a hybrid room through a radiation monitoring application where a virtual representation of the radiation cone beam, main X-ray scattering direction and dose distribution deposited on the surface of the patient are displayed on the hand-held screen.