Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation

Petter Törnberg

Abstract

Large language models (LLMs) are increasingly used for automated text annotation in tasks ranging from academic research to content moderation and hiring. Across 19 LLMs and two experiments totaling more than 4 million annotation judgments, we show that subtle identity cues embedded in text systematically bias annotation outcomes in ways that mirror racial stereotypes. In a names-based experiment spanning 39 annotation tasks, texts containing names associated with Black individuals are rated as more aggressive by 18 of 19 models and more gossipy by 18 of 19. Asian names produce a bamboo-ceiling profile: 17 of 19 models rate individuals as more intelligent, while 18 of 19 rate them as less confident and less sociable. Arab names elicit cognitive elevation alongside interpersonal devaluation, and all four minority groups are consistently rated as less self-disciplined. In a matched dialect experiment, the same sentence is judged significantly less professional (all 19 models, mean gap $-0.774$), less indicative of an educated speaker ($-0.688$), more toxic (18/19), and more angry (19/19) when written in African American Vernacular English rather than Standard American English. A notable exception occurs for name-based hireability, where fine-tuning appears to overcorrect, systematically favoring minority-named applicants. These findings suggest that using LLMs as automated annotators can embed socially patterned biases directly into the datasets and measurements that increasingly underpin research, governance, and decision-making.

Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation

Abstract

Large language models (LLMs) are increasingly used for automated text annotation in tasks ranging from academic research to content moderation and hiring. Across 19 LLMs and two experiments totaling more than 4 million annotation judgments, we show that subtle identity cues embedded in text systematically bias annotation outcomes in ways that mirror racial stereotypes. In a names-based experiment spanning 39 annotation tasks, texts containing names associated with Black individuals are rated as more aggressive by 18 of 19 models and more gossipy by 18 of 19. Asian names produce a bamboo-ceiling profile: 17 of 19 models rate individuals as more intelligent, while 18 of 19 rate them as less confident and less sociable. Arab names elicit cognitive elevation alongside interpersonal devaluation, and all four minority groups are consistently rated as less self-disciplined. In a matched dialect experiment, the same sentence is judged significantly less professional (all 19 models, mean gap ), less indicative of an educated speaker (), more toxic (18/19), and more angry (19/19) when written in African American Vernacular English rather than Standard American English. A notable exception occurs for name-based hireability, where fine-tuning appears to overcorrect, systematically favoring minority-named applicants. These findings suggest that using LLMs as automated annotators can embed socially patterned biases directly into the datasets and measurements that increasingly underpin research, governance, and decision-making.
Paper Structure (1 section, 3 equations, 2 figures, 2 tables)

This paper contains 1 section, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction

Figures (2)

  • Figure 1: Annotation gap architecture across all tasks and ethnic groups. Each panel shows one minority group. Points represent individual model gap estimates (minority minus White yes-rate); the thick vertical line indicates the cross-model mean and shading shows $\pm$1 SD. Tasks are ordered by mean gap within each panel. Positive values indicate that the minority group is rated higher than White-named individuals; negative values indicate a penalty. Self-discipline (consistently at the lower end of all four panels) is the most uniformly penalized attribute across groups. Hireability (consistently at the upper end) is the most uniformly elevated. Arab names show the greatest within-task variation, reflecting simultaneous elevation on cognitive attributes and devaluation on interpersonal dimensions.
  • Figure 2: Universal dialect-based annotation gaps across all 19 models. AAVE minus SAE yes-rate gaps for five annotation outcomes: (A) professional tone, (B) educated speaker, (C) hireability, (D) perceived anger, and (E) toxicity. Models are ordered consistently across panels by their professional-tone gap (most negative at top). Horizontal bars show the per-model gap; error bars show approximate 95% confidence intervals. Bold vertical lines mark the cross-model mean ($\mu$); dashed lines mark zero. Bars are colored by organizational family. All 19 models show negative gaps for professional tone ($\mu = -0.77$), educated speaker ($\mu = -0.69$), and hireability ($\mu = -0.31$); all 19 show positive gaps for perceived anger ($\mu = +0.15$); and 18 of 19 show positive gaps for toxicity ($\mu = +0.09$). The uniform direction across models from nine organizations indicates that dialect-based devaluation of AAVE is a shared property of the current model generation.