Race, Ethnicity and Their Implication on Bias in Large Language Models

TL;DR

The paper tackles how race and ethnicity are internally represented in LLMs and how these representations influence bias across tasks. It introduces a mechanistic interpretability pipeline combining multi-class probing, neuron-level attribution, and targeted intervention to locate race-encoding directions and causal neurons, then tests these mechanisms on ToxiGen and C-REACT across three models. The findings show race information is distributed across multiple neurons and semantic facets, with stereotype associations embedded in pretraining but reused in task contexts; interventions can reduce bias but do not erase the underlying representations, revealing a behavioral rather than purely representational locus. The work highlights model- and task-specific pathways for bias, underscoring the need for targeted, architecture-aware mitigation strategies that go beyond post-hoc outcome corrections.

Abstract

Large language models (LLMs) increasingly operate in high-stakes settings including healthcare and medicine, where demographic attributes such as race and ethnicity may be explicitly stated or implicitly inferred from text. However, existing studies primarily document outcome-level disparities, offering limited insight into internal mechanisms underlying these effects. We present a mechanistic study of how race and ethnicity are represented and operationalized within LLMs. Using two publicly available datasets spanning toxicity-related generation and clinical narrative understanding tasks, we analyze three open-source models with a reproducible interpretability pipeline combining probing, neuron-level attribution, and targeted intervention. We find that demographic information is distributed across internal units with substantial cross-model variation. Although some units encode sensitive or stereotype-related associations from pretraining, identical demographic cues can induce qualitatively different behaviors. Interventions suppressing such neurons reduce bias but leave substantial residual effects, suggesting behavioral rather than representational change and motivating more systematic mitigation.

Figures (6)

• Figure 1: With MLP, we locate neurons relevant to race information and inspect them via Logit Lens. For the higher activation score for target race, we adjust its value to steer model's behavior.
• Figure 2: Mean activation values of race encoding neurons when processing text from each racial group (ToxiGen). Diagonal cells represent neurons processing their target group. Higher values (red) indicate stronger activation; lower values (blue) indicate weak, negative activations.
• Figure 3: Prompt template for race prediction on C-REACT indirect mentions.
• Figure 4: Correct prediction rates after neuron intervention across amplification factors. Direct neuron intervention (solid) generally outperforms Indirect intervention (dashed), demonstrating that neurons encoding explicit racial terminology have stronger causal influence on predictions.
• Figure 5: Prediction distribution after neuron intervention on misclassified samples. Direct intervention (solid bars) eliminates the original bias entirely (orange 'Original Prediction' bars = 0%) across all models and factors, while Indirect intervention (faded bars) leaves residual bias. Higher amplification factors increase Unknown responses (gray)