Debiasing LLMs by Masking Unfairness-Driving Attention Heads

TL;DR

This paper investigates why LLMs produce unfair outputs and reveals that Direct Answer prompting activates a small set of bias-causing attention heads, while Chain-of-Thought prompting redirects computation away from these heads. By comparing DA and CoT prompts, the authors develop DiffHeads, a lightweight, three-stage framework that identifies differential unfair heads via a per-head contribution score and masks them during inference, achieving substantial fairness improvements (around 40–50% on key models) with minimal impact on utility. The approach reframes bias mitigation as a mechanistic, head-level intervention that requires no retraining or external classifiers, and it provides an auditing signal through token-to-head contribution analysis. Practically, DiffHeads offers a deployment-friendly debiasing technique applicable to many LLMs, along with insights into how prompting styles influence internal mechanisms and bias pathways.

Abstract

Large language models (LLMs) increasingly mediate decisions in domains where unfair treatment of demographic groups is unacceptable. Existing work probes when biased outputs appear, but gives little insight into the mechanisms that generate them, leaving existing mitigations largely fragile. In this paper, we conduct a systematic investigation LLM unfairness and propose DiffHeads, a lightweight debiasing framework for LLMs. We first compare Direct-Answer (DA) prompting to Chain-of-Thought (CoT) prompting across eight representative open- and closed-source LLMs. DA will trigger the nature bias part of LLM and improve measured unfairness by 534.5%-391.9% in both one-turn and two-turn dialogues. Next, we define a token-to-head contribution score that traces each token's influence back to individual attention heads. This reveals a small cluster of bias heads that activate under DA but stay largely dormant with CoT, providing the first causal link between prompting strategy and bias emergence. Finally, building on this insight, we propose DiffHeads that identifies bias heads through differential activation analysis between DA and CoT, and selectively masks only those heads. DiffHeads reduces unfairness by 49.4%, and 40.3% under DA and CoT, respectively, without harming model utility.

Figures (6)

• Figure 1: Prompting approach shapes model fairness. A Direct Answer (DA) prompt elicits an unfair, gender-biased reply, while the prompt augmented with a Chain-of-Thought (CoT) cue prompts the model to generate a fair output.
• Figure 2: An intuitive example of one-turn and one-turn questions, based on the same fairness-sensitive question.
• Figure 3: Unfairness scores of Direct‑Answer(DA) and Chain‑of‑Thought (CoT) prompting approaches across one-turn and two-turn conversation settings. (a)–(b) Open‑source models including Gemma-3-12B-It (Gemma‑3), Llama-3.1-8B-Instruct (Llama‑3.1), Qwen2.5-7B-Instruct (Qwen‑2.5), DeepSeek-V2-Lite-Chat (DS-V2) on one‑turn and two‑turn conversation settings. (c)–(d) Closed‑source models including GPT‑4o‑mini, Grok‑3, Claude-4-Sonnet (Claude‑4S), o4‑mini.
• Figure 4: Impact of incrementally masking attention heads on Qwen-2.5, measured using $|\Delta|_{\text{Unfairness}}$, with two 100-sample subsets—one with fair answers and one with unfair answers for DA and CoT under the one-turn conversation setting.
• Figure 5: DiffHeads. (a) Differential Response Set Generation. Direct‑Answer (DA) and Chain‑of‑Thought (CoT) prompts for the same question pool yield fair and unfair answer sets. (b) Differential Unfair Heads Identification. Attention heads are ranked on each set; those appearing in the top‑$k$ list for unfair answers but not for fair answers are collected as differ‑unfair heads. (c) Attention Head Masking. Zeroing these identified biased heads during decoding de-biases the LLM.