Between fact and fairy: tracing the hallucination metaphor in AI discourse

Abstract

Large and powerful language models such as OpenAI’s GPT model family, Google’s LaMDA and BERT or Meta’s LlaMA are integral to many applications, such as translation, summarization or language generation. They have become an inherent part of current everyday activities and working practices. These models produce and process language in an impressively convincing human-like manner, but also repeatedly generate outputs that appear untrustworthy and factually incorrect. In computer science (Ji et al. 2023) and popular discourse alike this phenomenon is called hallucination. The term is used broadly to describe various forms of untruthfulness, from factual errors to inconsistencies between prompt and output. This article discusses the hallucination metaphor guided by STS perspectives and takes software documentation as its main corpus of analysis. We examine model papers and documentation from leading tech companies to trace the hallucination metaphor and the discursive work it does. We claim that tech companies anthropomorphize the models, relieving them and themselves from responsibility over non-factual outputs by normalizing the use of the metaphor. Models are relegated to two main positions: either a learning child that needs time to develop or an illogic agent, a position that we connect to cultural scripts of madness.