Enhancing Breast Lesion Detection in Mammograms via Transfer Learning

Abstract

Early detection of breast cancer via mammography enhances patient survival rates, prompting this study to assess object detection models—Cascade R-CNN, YOLOv12 (S, L, and X variants), RTMDet-X, and RT-DETR-X—for detecting masses and calcifications across four public datasets (INbreast, CBIS-DDSM, VinDr-Mammo, and EMBED). The evaluation employs a standardized preprocessing approach (CLAHE, cropping) and augmentation (rotations, scaling), with transfer learning tested by training on combined datasets (e.g., INbreast + CBIS-DDSM) and validating on held-out sets (e.g., VinDr-Mammo). Performance is measured using precision, recall, mean Average Precision at IoU 0.5 ((Formula presented.)), and F1-score. YOLOv12-L excels in mass detection with an (Formula presented.) of 0.963 and F1-score up to 0.917 on INbreast, while RTMDet-X achieves an (Formula presented.) of 0.697 on combined datasets with transfer learning. Preprocessing improves (Formula presented.) by up to 0.209, and transfer learning elevates INbreast performance to an (Formula presented.) of 0.995, though it incurs 5–11% drops on CBIS-DDSM (0.566 to 0.447) and VinDr-Mammo (0.59 to 0.5) due to domain shifts. EMBED yields a low (Formula presented.) of 0.306 due to label inconsistencies, and calcification detection remains weak ((Formula presented.) < 0.116), highlighting the value of high-capacity models, preprocessing, and augmentation for mass detection while identifying calcification detection and domain adaptation as key areas for future investigation.