Superimposing holograms on real world objects using HoloLens 2 and its depth camera

Abstract

Augmented Reality glasses such as HoloLens 2 may provide visual guidance during surgical interventions. To superimpose the holograms on real world objects (RWO), for instance a patient, spatial registration is required. In this work, we propose an approach to automatically register a hologram to the according RWO. To this end, the framework utilizes the depth camera of HoloLens 2 to acquire the point cloud (PC) of the RWO. A novel and recently published PC registration algorithm allows to register the PC of the RWO and the hologram after a rough initial placement without any need for pre-processing or outlier removal. The approach is evaluated by measuring displacements between certain known positions of the hologram and the RWO. The first metric relies on measuring points using an optically tracked stylus while the second is based on visually perceived positions. The median displacements were 22.3 mm, 35.6 mm, and 13.3 mm for the x-, y-, and z-axes in the first metric and 8.1 mm, 4.3 mm, and 11.9 mm for the second metric. Even though the accuracy is not yet adequate for many surgical interventions, the framework provides an initial step for a convenient marker less registration of holograms to an RWO.