BC-FND: An Approach Based on Hierarchical Bilinear Fusion and Multimodal Consistency for Fake News Detection

Abstract

Fake news with multimedia on social media is deceptive, widely spread, and has serious negative effects. Therefore, multimodal fake news detection has become a popular and extensively studied topic. However, the existing methods have two shortcomings. 1) Different types of extractors are used for text and images, making it difficult to align the extracted features to the same embedding space. 2) The complex fusion approach leads to an increase in the number of features and parameters that generate redundancy and noise easily. To address these problems, we propose a simple yet powerful multimodal fake news detection model (BC-FND). It utilizes contrastive learning of CLIP to align textual and visual features to the same embedding space while using a consistency loss function to learn consistency between real news text and images as well as inconsistency between fake news text and images. Additionally, BERT is employed for extracting semantic and contextual information from text while a hierarchical bilinear fusion network is designed to achieve full complementarity between textual and visual features. Cross-entropy and consistency loss functions jointly optimize BC-FND for improved accuracy in detecting fake news. We also introduce the Weibo23 dataset which is more challenging since it's closer to the real social media environment. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods on two public datasets and the Weibo23 dataset.