HAMF: A Novel Hierarchical Attention-Based Multi-Modal Fusion Model for Parkinson’s Disease Classification and Severity Prediction

Abstract

Parkinson’s Disease (PD) is a neurodegenerative disorder that requires correct diagnosis and continuous monitoring of the disease severity. The state-of-the-art methods tend to be unimodal or lack robustness in generalizing between modalities, and hence cannot be applied clinically in diverse populations. A comprehensive approach is a multi-modal framework that overcomes these limitations by integration of brain Magnetic Resonance Imaging (MRI) data, gait analysis, and speech signals for enhanced classification and severity estimation of PD. A Hierarchical Attention-based Multi-modal Fusion (HAMF) model is developed in this paper to employ hierarchical attention mechanism at feature and decision levels to help the model learn representations at various levels. This leads to richer feature extraction, besides fusing different data modalities with accurate integration. Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) methods are used in optimizing the model, by which the convergence speed raised by 15–20 %. An accuracy of 94.2 % was achieved, thus improving by 4–5 %, compared to the existing methodologies. Temporal Convolutional Network (TCN) which can capture long-range temporal dependencies, was used in the longitudinal severity estimation task, achieving a Mean Squared Error (MSE) of 0.12 in disease progression forecasting. Beyond this, Domain-Adversarial Neural Network (DANN) enables improved cross-domain generalization and maintains a consistent classification accuracy of 90-93% on diversified datasets. Finally, SHapley Additive exPlanations - Class Activation Maps (SHAP-CAM) further enhanced the model explainability. During the conduct of this work, 85% of all cases provided clinically interpretable insights that allowed clinicians to conduct personalized treatment planning in a more robust and interpretable way. This work substantially extends current multi-modal diagnosis and analysis of PD progression by offering a robust and interpretable tool to clinicians for personalized treatment planning.