Gate-Fusion Transformer for Multimodal Sentiment Analysis

Abstract

Computational analysis of human multimodal sentiment is an emerging research area. Fusing semantic, visual and acoustic modalities requires exploring the inter-modal and intra-modal interactions. The first challenge for the inter-modal understanding is to break the heterogeneous gap between different modalities. Meanwhile, when modeling the intra-modal connection of time-series data, we must deal with the long-range dependencies among multiple steps. Moreover, The time-series data usually is unaligned between different modalities because individually specialized processing approaches or sampling frequencies. In this paper, we propose a method based on the transformer and the gate mechanism - the Gate-Fusion Transformer - to address these problems. We conducted detailed experiments for verifying the effectiveness of our proposed method. Because of the flexibility of gate-mechanism for information flow controlling and the great modeling power of the transformer for modeling the inter- and intra-modal interactions, we can achieve superior performance compared with the current state-of-the-art method but more extendible and flexible by stacking multiple gate-fusion blocks.