Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?

TL;DR

The paper investigates whether demographic bias in LLMs can be mitigated via inference-time, mechanistic interventions without erasing core demographic recognition. It uses a multi-task framework with sparse autoencoder–derived features to separate causally relevant demographic recognition from stereotype-driven associations, comparing attribution- and correlation-based ablations across Demo-R and Demo-L prompts. Key findings show that bias reduction is largely dimension- and task-specific: attribution-based ablations best reduce gender and race bias while preserving name recognition, whereas education bias is better tackled with correlation-based ablations, though some interventions induce prior collapse. These results highlight the contextual nature of bias encoding and suggest surgical debiasing is feasible but requires carefully matched interventions to the specific bias mechanisms and task context. The work underscores the importance of mechanistic interpretability for safe, interpretable debiasing in real-world NLP systems, and it points to future work on validating across languages and model families and on extending to broader demographic dimensions.

Abstract

We investigate how independent demographic bias mechanisms are from general demographic recognition in language models. Using a multi-task evaluation setup where demographics are associated with names, professions, and education levels, we measure whether models can be debiased while preserving demographic detection capabilities. We compare attribution-based and correlation-based methods for locating bias features. We find that targeted sparse autoencoder feature ablations in Gemma-2-9B reduce bias without degrading recognition performance: attribution-based ablations mitigate race and gender profession stereotypes while preserving name recognition accuracy, whereas correlation-based ablations are more effective for education bias. Qualitative analysis further reveals that removing attribution features in education tasks induces ``prior collapse'', thus increasing overall bias. This highlights the need for dimension-specific interventions. Overall, our results show that demographic bias arises from task-specific mechanisms rather than absolute demographic markers, and that mechanistic inference-time interventions can enable surgical debiasing without compromising core model capabilities.

Figures (7)

• Figure 1: The pipeline implements bidirectional evaluation using two prompt formats: Demo-R where demographic labels appear after items ("Jack -White"), and Demo-L where labels precede items ("Asian -Ming"). Following structured prediction generation, we apply attribution analysis and correlation analysis using sparse autoencoder (SAE) features to identify neural representations responsible for demographic associations. These features inform targeted ablation experiments to test their causal roles in both legitimate recognition and stereotyping behaviors.
• Figure 2: Each panel shows the percentage change from baseline performance when applying different ablation methods. Points are colored by the source task being ablated and shaped by ablation method type. The x-axis reports change in name prediction accuracy, and the y-axis reports change in profession bias (KL divergence). The bottom-right green region represents the ideal outcome, where ablations improve accuracy ($\uparrow$) while reducing bias ($\downarrow$). The top-left red region reflects the worst case, with accuracy loss ($\downarrow$) and increased bias ($\uparrow$). The yellow regions indicate trade-offs, where one improves while the other worsens. For example, an orange square on the left panel at (+0.75%, -6%) indicates a 0.75% improvement in name prediction accuracy and a 6% reduction in profession bias, corresponding to ablating gender–name attribution features.
• Figure 3: Joint effects on education profession bias (x-axis) and demographic profession biases (y-axis), both measured by KL divergence. The bottom-left green region represents the ideal outcome where ablations reduce both biases ($\downarrow$). The top-right red region reflects the worst case, with increases in both biases ($\uparrow$). The yellow regions indicate trade-offs, where one bias decreases while the other worsens. Multiple data points per ablation method reflect results from ablating different profession tasks (education, gender, or race).
• Figure 4: Profession-specific impact of attribution feature ablation on gender and race bias. Each bar shows the KL divergence from uniform distribution for individual professions, with baseline performance (blue bars) and post-ablation performance (orange bars). The $\Delta$ values indicate the change after ablation: negative values (green) indicate unchanged or reduced bias (closer to reference), while positive values (red) indicate increased bias. Left panel: Gender-Name attribution ablation shows minimal profession-specific variation, with most professions unchanged. Right panel: Race-Name attribution ablation demonstrates heterogeneous effects across professions, with substantial bias reductions for technical roles.
• Figure D.5: Each panel shows the percentage change from baseline performance when applying different ablation methods. Points are colored by the source task being ablated and shaped by ablation method type. The x-axis reports change in name prediction accuracy, and the y-axis reports change in profession bias (KL divergence). The bottom-right green region represents the ideal outcome, where ablations improve accuracy ($\uparrow$) while reducing bias ($\downarrow$). The top-left red region reflects the worst case, with accuracy loss ($\downarrow$) and increased bias ($\uparrow$). The yellow regions indicate trade-offs, where one improves while the other worsens.