HFT-Net: Hybrid Fusion Transformer Network for Multi-Source Breast Cancer Classification

Abstract

Automatic classification of breast cancer histopathological images presents significant challenges due to morphological ambiguities between disease subtypes, significant tissue heterogeneity, and limited availability of high-quality labeled datasets. We propose HFT-Net, a hybrid deep learning model that addresses the limitations of traditional single-architecture approaches in generalizing complex visual patterns. Unlike conventional feature fusion methods, our model employs a multi-head attention mechanism (MHA) to enrich information interaction and learn meaningful feature relationships, creating more discriminative representations. Despite the large data requirement of deep models, transfer learning and fine-tuning techniques enabled high success with few samples, and an efficient learning process was achieved by adapting pre-trained models. HFT-Net, which was developed as a solution to the problem of low generalization capacity of models optimized for a single dataset, aims for balanced performance and consistent results were obtained in three different datasets. As a result of extensive experimental evaluations, the proposed model shows competitive performance with 95.08% accuracy and 0.95 F1-score on the 8-class BreakHis dataset, 92.00% accuracy on the BACH secret test dataset, 92.50% accuracy and 0.92 F1-score on the BACH dataset, and 58.07% accuracy and 0.58 F1-score on the BRACS dataset.