Which Demographics do LLMs Default to During Annotation?

TL;DR

The paper investigates whether large language models exhibit a default demographic persona when used for subjective text annotation, by comparing socio-demographic prompts, placebo prompts, and non-demographic prompts on the POPQUORN dataset with two models (GPT-4o and Claude). It finds measurable but nuanced demographic effects, with race and age showing notable influences and politeness being more responsive to demographic cues than offensiveness. Placebo prompts largely fail to produce consistent attribute-specific shifts, while socio-demographic prompting yields systematic but model-dependent changes, and results vary across tasks and models. The work highlights persistent biases in LLM-driven annotation and argues for bias-aware design and robust measurement tools as critical for equitable and reliable data labeling in practice.

Abstract

Demographics and cultural background of annotators influence the labels they assign in text annotation -- for instance, an elderly woman might find it offensive to read a message addressed to a "bro", but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not under-represent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts to when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., "you are an annotator who lives in house number 5") to demographics-conditioned prompts ("You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}"). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects.

Figures (1)

• Figure 1: Our objective is to identify which human demographic groups are mimicked by LLMs during subjective annotation tasks on text data.