Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularization

TL;DR

The paper tackles bias in encoder LMs and the risk that downstream fine-tuning reintroduces bias, proposing Low Variance Regularization (LVR) to debias embeddings without attribute labels. By enforcing class-wise embedding variance through class centers and a center-informed total loss $\mathcal{L}_{total} = \mathcal{L}_t + \lambda \mathcal{L}_r + \mathcal{L}_c$, the method aims to reduce distributional shift while preserving discriminative power. Experiments with BERT-Base and RoBERTa-Base plus adapters on BIOS, FCDL18, and PAN16 show that AdpLVR generally reduces protected-attribute leakage more effectively than supervised baselines, with competitive or improved task performance in several settings. This approach offers scalable debiasing for unseen attributes without requiring labels, potentially broadening practical robustness of encoder-based transformers in diverse downstream tasks, though it relies on a binary gender definition and assumes classification Tasks and adapter-based fine-tuning.

Abstract

Language models frequently inherit societal biases from their training data. Numerous techniques have been proposed to mitigate these biases during both the pre-training and fine-tuning stages. However, fine-tuning a pre-trained debiased language model on a downstream task can reintroduce biases into the model. Additionally, existing debiasing methods for downstream tasks either (i) require labels of protected attributes (e.g., age, race, or political views) that are often not available or (ii) rely on indicators of bias, which restricts their applicability to gender debiasing since they rely on gender-specific words. To address this, we introduce a novel debiasing regularization technique based on the class-wise variance of embeddings. Crucially, our method does not require attribute labels and targets any attribute, thus addressing the shortcomings of existing debiasing methods. Our experiments on encoder language models and three datasets demonstrate that our method outperforms existing strong debiasing baselines that rely on target attribute labels while maintaining performance on the target task.