Multimodal image registration with deep context reinforcement learning

Abstract

Automatic and robust registration between real-time patient imaging and pre-operative data (e.g. CT and MRI) is crucial for computer-aided interventions and AR-based navigation guidance. In this paper, we present a novel approach to automatically align range image of the patient with pre-operative CT images. Unlike existing approaches based on the surface similarity optimization process, our algorithm leverages the contextual information of medical images to resolve data ambiguities and improve robustness. The proposed algorithm is derived from deep reinforcement learning algorithm that automatically learns to extract optimal feature representation to reduce the appearance discrepancy between these two modalities. Quantitative evaluations on 1788 pairs of CT and depth images from real clinical setting demonstrate that the proposed method achieves the state-of-the-art performance.