Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models

TL;DR

The paper investigates whether large language models (LLMs) exhibit partisan biases when used as annotators for political text. It replicates a human study by applying party cues to immigration statements and analyzes sentiment labels with an ordered logistic regression model $y_{ijk} = cue_j + content_i + llm_k + \epsilon_{ijk}$. Findings show that left-leaning cues (e.g., Green, SPÖ) push labels toward positive sentiment while right-leaning cues (e.g., FPÖ, ÖVP) push toward negative sentiment, with effects that are larger and less consistent across models than in humans. These results highlight the need for validation and caution when employing LLMs as political text annotators, given training-data priors and model-specific biases that can affect replicability and interpretation.

Abstract

Human coders are biased. We test similar biases in Large Language Models (LLMs) as annotators. By replicating an experiment run by Ennser-Jedenastik and Meyer (2018), we find evidence that LLMs use political information, and specifically party cues, to judge political statements. Not only do LLMs use relevant information to contextualize whether a statement is positive, negative, or neutral based on the party cue, they also reflect the biases of the human-generated data upon which they have been trained. We also find that unlike humans, who are only biased when faced with statements from extreme parties, LLMs exhibit significant bias even when prompted with statements from center-left and center-right parties. The implications of our findings are discussed in the conclusion.

Figures (3)

• Figure 1: Within-LLM consistency.
• Figure 2: Inter-coder reliability scores within-LLM family, and across all LLMs. ICR scores estimated for each party cue.
• Figure 3: Inter-coder reliability scores between LLMs and human annotators. On the left, within-model discrepancies across runs are adjudicated using majority rules. On the right, the labels from all runs from every LLM is compared to the labels from human annotators.