Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpreting Bias in Large Language Models: A Feature-Based Approach

Nirmalendu Prakash, Lee Ka Wei Roy

TL;DR

This work tackles how gender bias propagates in large language models by introducing a feature-based bias analysis framework inspired by causal mediation. It defines a feature map that tracks bias-related latent concepts across layers and validates it with activation and attribution patching on models like LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3, using a professions dataset and counterfactual debiasing to study both origin and propagation pathways. The authors show that bias largely originates in early layers, propagates through residual streams and attention heads, and can be mitigated with targeted interventions at the origin or during propagation, though efficacy varies by model and bias form. Their results highlight the need for component- and context-specific debiasing strategies and lay groundwork for extending the approach to other biases and multilingual settings. Overall, the paper contributes a granular, mechanistic view of bias in LLMs that informs more precise and potentially safer debiasing methodologies.

Abstract

Large Language Models (LLMs) such as Mistral and LLaMA have showcased remarkable performance across various natural language processing (NLP) tasks. Despite their success, these models inherit social biases from the diverse datasets on which they are trained. This paper investigates the propagation of biases within LLMs through a novel feature-based analytical approach. Drawing inspiration from causal mediation analysis, we hypothesize the evolution of bias-related features and validate them using interpretability techniques like activation and attribution patching. Our contributions are threefold: (1) We introduce and empirically validate a feature-based method for bias analysis in LLMs, applied to LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3 with templates from a professions dataset. (2) We extend our method to another form of gender bias, demonstrating its generalizability. (3) We differentiate the roles of MLPs and attention heads in bias propagation and implement targeted debiasing using a counterfactual dataset. Our findings reveal the complex nature of bias in LLMs and emphasize the necessity for tailored debiasing strategies, offering a deeper understanding of bias mechanisms and pathways for effective mitigation.

Interpreting Bias in Large Language Models: A Feature-Based Approach

TL;DR

This work tackles how gender bias propagates in large language models by introducing a feature-based bias analysis framework inspired by causal mediation. It defines a feature map that tracks bias-related latent concepts across layers and validates it with activation and attribution patching on models like LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3, using a professions dataset and counterfactual debiasing to study both origin and propagation pathways. The authors show that bias largely originates in early layers, propagates through residual streams and attention heads, and can be mitigated with targeted interventions at the origin or during propagation, though efficacy varies by model and bias form. Their results highlight the need for component- and context-specific debiasing strategies and lay groundwork for extending the approach to other biases and multilingual settings. Overall, the paper contributes a granular, mechanistic view of bias in LLMs that informs more precise and potentially safer debiasing methodologies.

Abstract

Large Language Models (LLMs) such as Mistral and LLaMA have showcased remarkable performance across various natural language processing (NLP) tasks. Despite their success, these models inherit social biases from the diverse datasets on which they are trained. This paper investigates the propagation of biases within LLMs through a novel feature-based analytical approach. Drawing inspiration from causal mediation analysis, we hypothesize the evolution of bias-related features and validate them using interpretability techniques like activation and attribution patching. Our contributions are threefold: (1) We introduce and empirically validate a feature-based method for bias analysis in LLMs, applied to LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B-v0.3 with templates from a professions dataset. (2) We extend our method to another form of gender bias, demonstrating its generalizability. (3) We differentiate the roles of MLPs and attention heads in bias propagation and implement targeted debiasing using a counterfactual dataset. Our findings reveal the complex nature of bias in LLMs and emphasize the necessity for tailored debiasing strategies, offering a deeper understanding of bias mechanisms and pathways for effective mitigation.
Paper Structure (24 sections, 4 figures, 7 tables)

This paper contains 24 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bias Studies in NLP
  4. LLM Interpretability
  5. Experimental Setup
  6. Bias Analysis
  7. Where is Gender Bias Introduced?
  8. Does Patching Lead to Reversal of Profession Information?
  9. How does Bias Propagate?
  10. Does Patching Lead to Reversal of Profession Information?
  11. Role of MLPs at upper layers
  12. LLM Debiasing
  13. Other Forms of Gender Biases
  14. Conclusion and Future Work
  15. Ethics Statement
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Conceptualized feature map in LLMs for prompts: a) "The doctor said that" and b) "Washing the dishes is the duty of the".
  • Figure 2: Bias Propagation for the prompt "The doctor said that" on LLaMA-3-8B. Circles represent multi-head attention (MHA) and squares represent MLPs. Purple squares represent MLPs associated with the feature. Blue circles represent MHA responsible for copying and green lines show the corresponding connections. Digits on the left represent layer indexes.
  • Figure 3: (a) Fraction of samples where logit is reversed vs MLP layers patched (b) Fraction of samples where logit is reversed vs top K value.
  • Figure 4: Bias Propagation for the prompt Washing the dishes is the duty of the on LLaMA-3-8B. Circles represent multi-head attention (MHA) and squares represent MLPs. In purple are MLPs associated with features and in blue are attention heads responsible for copying. Green lines represent important connections. Digits on the left represent layer indexes.