A Practical Multimodal Fusion System With Uncertainty Modeling for Robust Visual and Affective Applications

Abstract

Multimodal fusion models are widely applied in practical tasks such as image classification and sentiment analysis. However, real-world data often contain noisy or imbalanced modality inputs, significantly affecting the performance of traditional fusion methods. To address this challenge, we propose an Uncertainty-Aware Multimodal Fusion (UAMF) framework designed to enhance robustness under noisy conditions. UAMF jointly models aleatoric uncertainty from noisy data and epistemic uncertainty from model limitations. It integrates a Dynamic Direct Computation (DDC) module for adaptive uncertainty estimation and an Uncertainty Fusion Mechanism (UFM) that fuses predictions using Bayesian updates. We evaluate the effectiveness of our method on two practical tasks: image classification (FOOD101, CUB-200) and multimodal sentiment analysis (MOSI, MOSEI). Compared with the baseline model QMF, UAMF achieves an average accuracy improvement of 2.3% across tasks (e.g., +3.1% on CUB-200 and +1.2% on MVSA-Single under Gaussian noise). In addition, it reduces inference time by 52%, from 187.2ms to 89.5ms, compared to the recent state-of-the-art model EAU. The results demonstrate that UAMF outperforms existing methods across both application domains, particularly under noisy or modality-impaired conditions. These findings highlight the robustness and generalizability of the proposed method in real-world multimodal learning scenarios.