Artificial intelligence versus journalists: The quality of automated news and bias by authorship using a Turing test*

Abstract

The integration of Artificial Intelligence (AI) in the media results in the publication of thousands of automated news articles in Spanish every day. This study uses a Turing test to compare the quality of news articles written by professional journalists (from Efe) with those produced by natural language generation (NLG) software (from Narrativa). Based on Sundar’s dimensions (1999) crucial to news perception – credibility, readability and journalistic expertise –, an internationally validated experimental methodology is employed, exploring a novel topic in Spanish: health information. The experiment deliberately varied real and declared authorships – AI and human journalists – to detect potential biases in assessing authorship credibility. A self-administered questionnaire adapted for online surveys was used (N=222), and gender imbalances were minimized to ensure gender equality in the sample (N=128). The study reveals that there are no significant differences between news articles generated by the AI and those written by professional journalists. Both types of news are considered equally credible, though some biases are detected in the evaluation of declared authorship: the AI author is perceived as more believable than the human, while the human journalist is perceived as creating a more lively narrative. The study concludes that it is feasible to produce automated news in Spanish without compromising its quality. In the global media landscape, automated systems employing NLG, machine learning and sophisticated databases successfully advance into new domains such as health information.