MBA: A Multimodal Bilinear Attention Model with Residual Connection for Abstractive Multimodal Summarization

Abstract

The combination of vision and natural language modalities has become an important topic in both computer vision and natural language processing research communities. Multimodal summarization has received unprecedented attention with the rapid growth of multimodal information. This paper proposes MBA which consists of pre-trained feature extractors, text encoder, image encoder, multimodal bilinear attention fusion module, and summary decoder to complete abstractive multimodal summarization task. A residual network is added to the model to enhance the textual modality information and alleviate the modality-bias problem. Experiments show that the model is better than the baseline models and performs better than text summarization methods that ignore visual modality.