A benchmark for multimodal fact verification with explainability through 5W Question-Answering

Abstract

Combating disinformation is one of the burning societal crises - about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Disinformation can manipulate democracy, public opinion, disrupt markets, and cause panic or even fatalities. Thus, swift detection and possible prevention of disinformation are vital, especially with the daily flood of 3.2 billion images and 720,000 hours of videos on social media platforms, necessitating efficient fact verification. Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR), the research community lacks substantial effort in multimodal fact verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3 million samples that pushes the boundaries of the domain of fact verification via a multimodal fake news dataset, in addition to offering explainability through the concept of 5W question-answering. Salient features of the dataset are: (i) textual claims, (ii) GPT3.5-generated paraphrased claims, (iii) associated images, (iv) stable diffusion-generated additional images (i.e., visual paraphrases), (v) pixel-level image heatmap to foster image-text explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news stories.