Multimodal Deep Learning for Social Media Popularity Prediction with Attention Mechanism

Abstract

Social media popularity estimation refers to predict the post's popularity using multimodal contents. The prediction performance heavily relies on the feature extraction part and fully leveraging multimodal heterogeneous data is of a great challenge in the practical settings. Despite remarkable progress have been made, most of the previous attempts are restrained from the essentially limited property of the employed single modality. Inspired by the recent success of multimodal learning, we propose a novel multimodal deep learning framework for the popularity prediction task, which aims to leverage the complementary knowledge from different modalities. Moreover, an attention mechanism is introduced in our framework, with the goal to assign large weights to specified modalities during the training and inference phases. To empirically investigate the effectiveness and robustness of the proposed approach, we conduct extensive experiments on the 2020 SMP challenge. The obtained results show that the proposed framework outperforms related approaches.