Counteracting French Fake News on Climate Change Using Language Models

Abstract

The unprecedented scale of disinformation on the Internet for more than a decade represents a serious challenge for democratic societies. When this process is focused on a well-established subject such as climate change, it can subvert measures and policies that various governmental bodies have taken to mitigate the phenomenon. It is therefore essential to effectively identify and counteract fake news on climate change. To do this, our main contribution represents a novel dataset with more than 2300 articles written in French, gathered using web scraping from all types of media dealing with climate change. Manual labeling was performed by two annotators with three classes: “fake”, “biased”, and “true”. Machine Learning models ranging from bag-of-words representations used by an SVM to Transformer-based architectures built on top of CamemBERT were built to automatically classify the articles. Our results, with an F1-score of 84.75% using the BERT-based model at the article level coupled with hand-crafted features specifically tailored for this task, represent a strong baseline. At the same time, we highlight perceptual properties as text sequences (i.e., fake, biased, and irrelevant text fragments) at the sentence level, with a macro F1 of 45.01% and a micro F1 of 78.11%. Based on these results, our proposed method facilitates the identification of fake news, and thus contributes to better education of the public.