Transfer Learning with Joint Optimization for Label-Efficient Medical Image Anomaly Detection

Abstract

Many medical imaging applications require robust capabilities for automated image anomaly detection. Supervised deep learning approaches can be employed for such tasks, but poses large data collection and annotation burdens. To address this challenge, recent works have proposed advanced unsupervised, semi-supervised or transfer learning based deep learning methods for label-efficient image anomaly detection. However, these methods often require extensive hyperparameter tuning to achieve good performance, and have yet to be demonstrated in data-scarce domain centric applications with nuanced normal-vs-anomaly distinctions. Here, we propose a practical label-efficient anomaly detection method that employs fine-tuning of pre-trained model based on a small target domain dataset. Our approach employs a joint optimization framework to enhance discriminative power for anomaly detection performance. In evaluations on two benchmark medical image datasets, we demonstrate (a) strong performance gains over state-of-the-art baselines and (b) increased label efficiency over standard fine-tuning approaches. Importantly, our approach reduces the need for large annotated datasets, requires minimal hyperparameter tuning, and shows stronger performance boost for more challenging anomalies (Supplement: http://s000.tinyupload.com/?file_id=24916959421870989415).