FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering

TL;DR

FairSteer introduces an inference-time debiasing framework that does not require retraining or prompt redesign. By demonstrating that fairness-related features are linearly separable in intermediate transformer activations, it computes Debiasing Steering Vectors from a small set of contrast prompts and dynamically applies activation steering during inference via BAD and DAS. Across six open-source LLMs and tasks spanning QA, counterfactual input evaluation, and open-ended generation, FairSteer reduces bias while largely preserving language modeling and knowledge-task performance, offering a practical, low-overhead bias mitigation solution. The work also provides insights into layer selection, data requirements, and the interplay between bias mitigation and model utility, with code to enable reproducibility.

Abstract

Large language models (LLMs) are prone to capturing biases from training corpus, leading to potential negative social impacts. Existing prompt-based debiasing methods exhibit instability due to their sensitivity to prompt changes, while fine-tuning-based techniques incur substantial computational overhead and catastrophic forgetting. In this paper, we propose FairSteer, a novel inference-time debiasing framework without requiring customized prompt design or model retraining. Motivated by the linear representation hypothesis, our preliminary investigation demonstrates that fairness-related features can be encoded into separable directions in the hidden activation space. FairSteer operates in three steps: biased activation detection, debiasing steering vector (DSV) computation, and dynamic activation steering. Specifically, it first trains a lightweight linear classifier to detect bias signatures in activations, and then computes DSVs as intervention directions derived from small contrastive prompt pairs. Subsequently, it performs debiasing by adjusting activations with DSVs in the inference stage. Comprehensive evaluation with six LLMs demonstrates the superiority of FairSteer across question-answering, counterfactual input evaluation and open-ended text generation tasks. Code will be released.

Figures (15)

• Figure 1: Overview of the FairSteer framework. FairSteer operates in three steps: (a) Biased Activation Detection: train linear classifiers to detect bias signature. (b) Debiasing Steering Vector Computation: compute the DSV by averaging the differences between activations corresponding to biased and unbiased prompts. (c) Dynamic Activation Steering: apply conditional interventions in the selected layer.
• Figure 2: Accuracy on validation set across different layers of LLMs.
• Figure 3: 2-D PCA visualization of biased and unbiased activations from the 14th layer of Vicuna-13B. The arrow represents the mean difference between biased and unbiased activations, corresponding to the DSV.
• Figure 4: Contrast prompt pairs example.
• Figure 5: Layer selection based on accuracy across six models.