Learning CNNs with pairwise domain adaption for real-time 6DoF ultrasound transducer detection and tracking from X-ray images

Abstract

In this paper, we present a fully automatic and real-time CNN-based system that achieves highly accurate and robust 6DoF pose estimation and tracking of Transesophageal Echocardiography (TEE) transducer from 2D X-ray images, a key enabler for integrating ultrasound and fluoroscopic image guidance in hybrid operating rooms for catheter-based procedures. Lightweight hierarchical CNNs are first pre-trained purely on a large number of synthetically-generated X-ray images with known ground truth poses. The pre-trained CNNs are then refined for generalization using only a small number of real X-ray images with annotated poses via our proposed pairwise domain adaptation scheme. To resolve the pose ambiguity caused by the self-symmetry of the TEE transducer and the translucent nature of X-ray imaging, a CNN classifier is trained to classify a correct pose from its flipped counterpart by seeing a large number of synthetically-generated pairs. The proposed system is validated on 1,663 fluoroscopic images from clinical studies, and achieves an error rate of 6.53% with a clinically relevant criteria (i.e., Projected Target Registration Error larger than 2.5 mm) and a frame rate of 83.3 frames per second in tracking mode, outperforming the state-of-the-art methods in terms of both accuracy and speed.