The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net

Abstract

We investigate the potential contribution of an AI system as a safety net application for radiologists in breast cancer screening. As a safety net, the AI alerts on cases suspected to be malignant which the radiologist did not recommend for a recall. We analyzed held-out data of 2,638 exams enriched with 90 missed cancers. In screening mammography settings, we show that a system alerting on 11 out of every 1,000 cases, could detect up to 10.7% of the radiologists’ missed cancers. Thus, significantly increasing radiologist’s sensitivity to 80.3%, while only slightly decreasing their specificity to 95.3%. Importantly, the safety net demonstrated a significant contribution to their performance even when radiologists utilized both mammography and ultrasound images. In those settings, it would have alerted 8.5 times per 1,000 cases, and detected 11.7% of the radiologists’ missed cancers. In an analysis of the missed cancers by an expert, we found that most of the cancers detected by the AI were visible post-hoc. Finally, we performed a reader study with five radiologists over 120 exams, 10 of which were originally missed cancers. The AI safety net was able to assist 3 out of the 5 radiologists in detecting missed cancers without raising any false alerts.