Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Bias Representations in Llama 2 Chat via Activation Steering

Dawn Lu, Nina Rimsky

TL;DR

This work probes societal bias in Llama 2 7B Chat using activation steering (Contrastive Activation Addition) to red-team bias across gender, race, and religion. It builds steering vectors from StereoSet and GPT-4-generated prompts and evaluates responses under original and steered conditions. Key findings include persistent gender bias after RLHF, model refusals on race and religion prompts, and evidence that RLHF increases cross-bias similarity, reducing nuanced distinctions between bias types. The study offers practical red-teaming guidance, highlighting the role of a refusal vector in eliciting biased behavior for robust evaluation and informing mitigation strategies.

Abstract

We address the challenge of societal bias in Large Language Models (LLMs), focusing on the Llama 2 7B Chat model. As LLMs are increasingly integrated into decision-making processes with substantial societal impact, it becomes imperative to ensure these models do not reinforce existing biases. Our approach employs activation steering to probe for and mitigate biases related to gender, race, and religion. This method manipulates model activations to direct responses towards or away from biased outputs, utilizing steering vectors derived from the StereoSet dataset and custom GPT4 generated gender bias prompts. Our findings reveal inherent gender bias in Llama 2 7B Chat, persisting even after Reinforcement Learning from Human Feedback (RLHF). We also observe a predictable negative correlation between bias and the model's tendency to refuse responses. Significantly, our study uncovers that RLHF tends to increase the similarity in the model's representation of different forms of societal biases, which raises questions about the model's nuanced understanding of different forms of bias. This work also provides valuable insights into effective red-teaming strategies for LLMs using activation steering, particularly emphasizing the importance of integrating a refusal vector.

Investigating Bias Representations in Llama 2 Chat via Activation Steering

TL;DR

This work probes societal bias in Llama 2 7B Chat using activation steering (Contrastive Activation Addition) to red-team bias across gender, race, and religion. It builds steering vectors from StereoSet and GPT-4-generated prompts and evaluates responses under original and steered conditions. Key findings include persistent gender bias after RLHF, model refusals on race and religion prompts, and evidence that RLHF increases cross-bias similarity, reducing nuanced distinctions between bias types. The study offers practical red-teaming guidance, highlighting the role of a refusal vector in eliciting biased behavior for robust evaluation and informing mitigation strategies.

Abstract

We address the challenge of societal bias in Large Language Models (LLMs), focusing on the Llama 2 7B Chat model. As LLMs are increasingly integrated into decision-making processes with substantial societal impact, it becomes imperative to ensure these models do not reinforce existing biases. Our approach employs activation steering to probe for and mitigate biases related to gender, race, and religion. This method manipulates model activations to direct responses towards or away from biased outputs, utilizing steering vectors derived from the StereoSet dataset and custom GPT4 generated gender bias prompts. Our findings reveal inherent gender bias in Llama 2 7B Chat, persisting even after Reinforcement Learning from Human Feedback (RLHF). We also observe a predictable negative correlation between bias and the model's tendency to refuse responses. Significantly, our study uncovers that RLHF tends to increase the similarity in the model's representation of different forms of societal biases, which raises questions about the model's nuanced understanding of different forms of bias. This work also provides valuable insights into effective red-teaming strategies for LLMs using activation steering, particularly emphasizing the importance of integrating a refusal vector.
Paper Structure (10 sections, 13 figures)

This paper contains 10 sections, 13 figures.

Table of Contents

  1. Introduction
  2. Methodology and Data
  3. Activation Clustering
  4. Steered Responses
  5. Original Responses
  6. Bias-steered responses
  7. Relationship Between Steering Vectors
  8. How are Bias and Refusal Related?
  9. How are Different Forms of Bias Related to Each Other?
  10. Conclusion

Figures (13)

  • Figure 1: Example of A/B contrast prompt used to generate steering vectors.
  • Figure 2: Gender (n=72)
  • Figure 3: Race (n=300)
  • Figure 4: Religion (n=78)
  • Figure 6: Prompts for Evaluation
  • ...and 8 more figures