BiasEdit: Debiasing Stereotyped Language Models via Model Editing

TL;DR

BiasEdit tackles stereotyped biases in large language models by introducing editor networks that perform localized parameter edits to remove biased associations while preserving language modeling performance. It defines a debiasing objective based on symmetric KL divergences between stereotyped and anti-stereotyped contexts, complemented by a retention loss to keep meaningless-context probabilities stable, forming the overall loss $\mathcal{L}_E = \mathcal{L}_d + \lambda \mathcal{L}_r$. Empirically, BiasEdit achieves state-of-the-art debiasing on StereoSet and Crows-Pairs across multiple models, with minimal degradation to LM capabilities and strong robustness to gender reversal and semantic generality. The work also introduces bias tracing to locate bias-related activations and shows that editing upper network blocks can mitigate adverse effects, supporting practical deployment of off-the-shelf unbiased models.

Abstract

Previous studies have established that language models manifest stereotyped biases. Existing debiasing strategies, such as retraining a model with counterfactual data, representation projection, and prompting often fail to efficiently eliminate bias or directly alter the models' biased internal representations. To address these issues, we propose BiasEdit, an efficient model editing method to remove stereotypical bias from language models through lightweight networks that act as editors to generate parameter updates. BiasEdit employs a debiasing loss guiding editor networks to conduct local edits on partial parameters of a language model for debiasing while preserving the language modeling abilities during editing through a retention loss. Experiments on StereoSet and Crows-Pairs demonstrate the effectiveness, efficiency, and robustness of BiasEdit in eliminating bias compared to tangental debiasing baselines and little to no impact on the language models' general capabilities. In addition, we conduct bias tracing to probe bias in various modules and explore bias editing impacts on different components of language models.

Figures (6)

• Figure 1: Debiasing a language model with BiasEdit.
• Figure 2: Debiasing a language model with BiasEdit. Editor networks $\phi$ are trained to produce edit shifts on partial parameters $\mathcal{W}$ of a language model while its parameters $\theta$ are frozen . After editing, an unbiased LM is obtained with the robustness of gender reversal and semantic generality. $\mathcal{L}_d$ and $\mathcal{L}_r$ refer to Equation \ref{['equation:2']} and \ref{['equation:3']} respectively. s: stereotyped. a: anti-stereotyped. m: meaningless.
• Figure 3: SS (%) and $\Delta$LMS (%) of debiased language models after editing the last layer in the MLP of different blocks. 1/2/3: the first/second/third block. 12: the first 2 blocks. 123: the first 3 blocks. -1/-2/-3, the last/penultimate/antepenultimate block, -321: the last 3 blocks. -21: the last 2 blocks.
• Figure 4: Gender Reversal Robustness. Pre-debias refers to SS of pre-trained language models on the gender reversal test set before debiasing. Debiased refers to SS of debiased models by BiasEdit.
• Figure 5: Gender bias tracing on GPT2-medium. (a) Comparing bias associations of bias attribute words on hidden states, attention layers, and MLP layers. (b) Comparing bias associations on single states of the bias attribute word, the token before the attribute term, and the attribute term. The bias impacts on output probability are mapped for the effect of (c-d) each hidden state on the context, (e-f) only MLP activations, and (g-h) only attention activations. * marks the corrupted bias attribute words and [] refers to the attribute terms in (c-h).