Adaptive Graph Learning with Multimodal Fusion for Emotion Recognition in Conversation

Abstract

Robust emotion recognition is a prerequisite for natural, fluid human–computer interaction, yet conversational settings remain challenging because emotions are shaped simultaneously by global topic flow and local speaker-to-speaker dependencies. Here, we introduce GASMER—Graph-Adaptive Structure for Multimodal Emotion Recognition—a unified architecture that tackles both issues. It uses the correlation structure based on graph neural networks (GNNs) to model the complex dependencies in the conversation, while adaptively learning the graph structure for GNNs. The experiments indicate that our model has strong performance that outperforms all existing graph-based approaches, and remains competitive when compared to recent multimodal fusion models, underscoring the importance of combining fine-grained multimodal fusion with adaptive graph learning for conversational emotion recognition. On the IEMOCAP dataset, GASMER improves accuracy by 2.7% and the weighted F1-score by 3.6% compared to the best baseline. On the MOSEI dataset, it achieves a 1.2% gain in binary classification accuracy (ACC-2).