Fact-Checking With Contextual Narratives: Leveraging Retrieval-Augmented LLMs for Social Media Analysis

Abstract

Fact-checking systems have gained traction as scalable solutions, yet they often face challenges such as handling diverse evidence sources, integrating multimodal data, and presenting comprehensive narratives. In this work, we propose cluster-based retrieval augmented verification with explanation (CRAVE), a novel framework that integrates retrieval-augmented large language models (LLMs) with clustering techniques to address multimodal misinformation on social media. The framework is designed to process multimodal inputs (text and images) and iteratively refine evidence through agent-based mechanisms. We validated the framework on multiple real-world and synthetic datasets, showing that breaking up evidence into narrative clusters improves both retrieval precision, clustering quality, and judgment accuracy, showcasing its potential as a robust decision-support tool for fact-checkers.