Uncovering the black box of medical image analysis algorithms: recent advances in explainable artificial intelligence in medical image analysis

Abstract

Artificial intelligence (AI) has evolved significantly since its inception in the mid-20th century. With the advent of powerful computational resources and the accumulation of large-scale datasets in the 21st century, deep learning has achieved groundbreaking advancements, leading to the development of models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), attention-based models, and diffusion models. One of the most transformative applications of AI is in medical imaging, where AI-driven techniques have significantly enhanced disease screening, diagnosis, treatment planning, and prognosis evaluation. Despite these advancements, AI models often function as “black boxes”, making them difficult to explain, thereby hindering their clinical adoption. The lack of transparency in AI-driven decision-making reduces trust among healthcare professionals, posing a critical barrier to widespread clinical implementation. To address this challenge, explainable AI (XAI) has emerged as a crucial research domain aimed at enhancing model explainability and trustworthiness. XAI techniques can be categorized based on explanation type (intrinsic vs. post-hoc), explanation specificity (model-specific vs. model-agnostic), and explanation scope (global vs. local). Intrinsic explainability is inherent in simpler models such as linear regression and decision trees, whereas post-hoc explainability is required for complex deep learning models. Model-specific explainability is tailored for particular architectures, while model-agnostic approaches, such as SHapley Additive exPlanations (SHAP) and local interpretable model-agnostic explanations (LIME), can be applied across different models. Global explanations provide insights into overall model behavior, whereas local explanations focus on individual predictions. Several XAI methods have been integrated into medical imaging analysis to enhance explainability. Feature attribution methods, such as deep learning important features (DeepLIFT) and occlusion sensitivity analysis, highlight critical regions in medical images that influence AI predictions. Saliency-based methods, including class activation mapping (CAM) and its extensions (Grad-CAM, Grad-CAM++, Score-CAM), visualize model attention in CNNs. Attention mechanisms, widely utilized in transformer-based architectures, provide enhanced feature weighting, improving model explainability. Prototype-based explanations identify representative cases within a dataset to facilitate human comprehension of AI decisions, while counterfactual explanations explore alternative scenarios to reveal causal relationships. Despite the progress in XAI research, several challenges remain. First, explanation quality varies across techniques, with some methods generating unreliable or inconsistent explanation. Second, computational costs associated with perturbation-based and sampling-intensive methods limit their real-time applicability. Third, a trade-off often exists between model accuracy and explainability, necessitating innovative solutions to balance both aspects. Finally, integrating XAI techniques into clinical workflows requires robust evaluation metrics that align with human-centered decision-making processes. Future research should focus on optimizing XAI algorithms for medical applications by reducing computational overhead, improving causal inference, and enhancing multimodal data integration to facilitate transparent and trustworthy AI-driven healthcare solutions.