Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads

TL;DR

The paper addresses the opaque internal mechanisms by which stereotypical biases are encoded in transformer PLMs. It introduces a gradient-based attention-head bias estimation framework that uses a per-head mask $m_{i}$ and the objective $\mathcal{L}_{|SEAT|}$ to compute head biases $b_{i,j}=\partial \mathcal{L}_{|SEAT|}/\partial m_{i,j}$, thereby identifying a small subset of biased attention heads. Through gender and racial bias analyses in BERT, GPT-2, and LLaMA-family models, and counter-stereotype experiments, the authors validate that these biased heads meaningfully contribute to stereotypes and can be mitigated via targeted masking of top biased heads, with minimal impact on language modeling. The study further demonstrates generalization to LLMs and suggests that instruction-tuning (RLHF) may reduce head biases. Overall, the work opens the black-box of PLM biases, provides a mechanistic framework for bias assessment, and offers a practical, low-cost direction for debiasing via selective head masking.

Abstract

Transformer-based pretrained large language models (PLM) such as BERT and GPT have achieved remarkable success in NLP tasks. However, PLMs are prone to encoding stereotypical biases. Although a burgeoning literature has emerged on stereotypical bias mitigation in PLMs, such as work on debiasing gender and racial stereotyping, how such biases manifest and behave internally within PLMs remains largely unknown. Understanding the internal stereotyping mechanisms may allow better assessment of model fairness and guide the development of effective mitigation strategies. In this work, we focus on attention heads, a major component of the Transformer architecture, and propose a bias analysis framework to explore and identify a small set of biased heads that are found to contribute to a PLM's stereotypical bias. We conduct extensive experiments to validate the existence of these biased heads and to better understand how they behave. We investigate gender and racial bias in the English language in two types of Transformer-based PLMs: the encoder-based BERT model and the decoder-based autoregressive GPT model. Overall, the results shed light on understanding the bias behavior in pretrained language models.

Figures (7)

• Figure 1: Bias score distributions for BERT-base gender (\ref{['fig:bias-score-distribution']}), GPT-2 gender (\ref{['fig:bias-score-distribution-gpt2']}), and BERT-base race (\ref{['fig:bias-score-distribution-racial']}).
• Figure 2: Attention head visualizations for BERT-base gender (\ref{['fig:biased-heads']}), GPT-2 gender (\ref{['fig:biased-heads-gpt2']}), BERT-base race (\ref{['fig:biased-heads-racial']}). Note that negative bias scores are converted to zero for better visual illustration.
• Figure 3: A running example for the counter-stereotype experiment. The four plots show the attention score (the boldface number) in the original sentence and the counter-stereotype sentence of a biased head (left two figures) and an unbiased head (right two figures). In this example, the target word is "emotional". The edge thickness is associated with its normalized attention score. BERT-base model is used in this example.
• Figure 4: Quantitative counter-stereotype experiments.
• Figure 5: Bias heads heatmap in prior debiased models. We highlight the top-5 anti-biased heads (with the largest negative bias scores) in red boxes in the original BERT-base and BERT-large.