Explainable multi-modal deep learning for transparent cancer diagnosis: integrating radiology, clinical features, and decision visualization

Abstract

Introduction: Although artificial intelligence–based cancer diagnostic models have demonstrated strong predictive performance, their lack of transparency and reliance on single-modality data continue to limit clinical trust and adoption. Effectively integrating multi-modal data with interpret-able decision-making remains a key challenge. Methods: We propose an explainable multi-modal deep learning framework that integrates radiological imaging and structured clinical features using attention-based fusion. Image-level explanations are generated using Grad-CAM++, while SHAP is employed to quantify clinical feature contributions, enabling unified and cross-modal aligned interpretation rather than independent uni-modal explanations. The framework was evaluated on publicly available datasets, including CBIS-DDSM mammography, Duke Breast Cancer MRI, and TCGA cohorts (BRCA, LUAD, and GBM), comprising a total of 3,842 images from 2,917 patients. Results: The proposed model consistently outperformed uni-modal approaches and simple fusion baselines, achieving an improved balance between sensitivity and specificity. Attention-based fusion demonstrated superior performance compared with feature concatenation, and the integration of explainability did not compromise predictive accuracy. Visual and clinical explanations highlighted diagnostically relevant tumor regions and established oncological risk factors. Stable performance across datasets indicates strong generalization capability. Discussion: These results demonstrate that explainable multi-modal learning can effectively combine accuracy, interpret-ability, and robustness, supporting the development of reliable AI-based decision-support systems for cancer diagnosis.