A Comparative Study in Large Language Models Usage for Fake News Detection

Abstract

This study presents a comparative analysis of the capabilities of Large Language Models (LLMs) in the automatic detection of fake news. The research focuses on evaluating the accuracy, precision, recall and F1-score of three state-of-the-art LLMs—OpenAI’s ChatGPT 4.0 beta, Meta’s Llama 3.1, and Google’s Gemini—using an extensive dataset of verified true and fake news. By employing a black-box testing method, the study categorizes the LLMs’ outputs and assesses their performance based on accuracy metrics. The results indicate moderate success in distinguishing between true and false news, with differences noted between the models and smaller, text classification specialized Natural Language Models (NLP), like Google‘s Bidirectional Encoder Representations from Transformers (BERT) model variants, trained on the same fake news dataset. The findings demonstrate the potential of LLMs as tools for combating misinformation, while also emphasizing the current limitations and the need for improvements in their accuracy and reliability. This paper provides insights into the challenges of utilizing LLMs for misinformation detection and highlights the importance of combining technological advancements with our distinct human cognition.