Robust multimodal breast cancer diagnosis using DeiT and ANN fusion with minimal features on imbalanced clinical-pathological data

Abstract

Breast cancer remains a leading cause of mortality and requires early and accurate diagnosis to improve patient outcomes. Traditional diagnostic methods often rely on either clinical records or imaging, limiting their ability to capture complementary information. This study proposes a deep learning-based multimodal framework that integrates pathological images with structured clinical data to improve breast cancer classification. We evaluated CNNs and transformer-based architectures (ViT, DeiT) on 3,764 paired pathological images and structured clinical data from 185 patients. We applied transfer learning, class weighting, SMOTE, and augmentation to address challenges of limited data and class imbalance. The DeiT-based multimodal model achieved 98% accuracy, 98% F1-score, and 100% AUC-ROC, outperforming both CNN- and transformer-only baselines. This work contributes a data-efficient multimodal framework that achieves state-of-the-art performance with limited data. It explicitly addresses class imbalance to enhance generalisation, and incorporates feature selection to improve interpretability. Moreover, it demonstrates that a lightweight DeiT-based model can outperform both CNNs and other transformers under data-scarce conditions.