LLMs as annotators: the effect of party cues on labelling decisions by large language models

Abstract

Human coders can be biased. We test whether Large Language Models (LLMs) replicate those biases when used as text annotators. By replicating an experiment conducted by Ennser-Jedenastik and Meyer (2018), we find evidence that LLMs use political information, and specifically party cues, to evaluate political statements. Not only do LLMs use relevant information to contextualize whether a statement is positive, negative, or neutral based on the party cue, they also reflect the biases of the human-generated data upon which they have been trained. We also find that unlike humans, who are only biased when faced with statements from extreme parties, some LLMs exhibit significant bias even when prompted with statements from center-left and center-right parties. The implications of our findings are discussed in the conclusion.