CroMe: Multimodal Fake News Detection Using Cross-Modal Tri-Transformer and Metric Learning

Abstract

Multimodal fake news detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modal relationships and integrating inter-modal similarities using advanced techniques. To address these issues, Cross-Modal Tri-Transformer and Metric Learning (CroMe) for multimodal fake news detection is proposed. CroMe utilizes bootstrapping language-image pre-training (BLIP) with frozen image encoders and large language models as encoders to capture detailed text, image, and combined image-text representations. The metric learning module employs a proxy anchor method to capture intra-modality relationships while the feature fusion module uses a Cross-Modal and Tri-Transformer for effective integration. The final fake news detector processes the fused features through a classifier to predict the authenticity of the content. Experiments on datasets show that CroMe excels in multimodal fake news detection.