Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences

TL;DR

This work tackles gender bias in pre-trained language models by leveraging high-quality, balanced sentences generated by large language models and filtering them through a causal-alignment mechanism. The core method, Fair-Gender, prompts an LLM via two steps to create pairwise attribute-target sentences, splits them into causally aligned ($X^{C}$) and unaligned ($X^{NC}$) sets, and uses a Structural Causal Model to estimate the causal effect of the aligned data on bias in PLMs. It then optimizes a joint objective $\ L = \L_b + \lambda \cdot L_r$ where $L_r$ preserves language modeling and $L_b$ enforces fairness with a KNN-based estimation of causal impact on aligned sentences. Experiments across BERT, ALBERT, and RoBERTa show Fair-Gender achieves strong fairness (SEAT, CrowS-Pairs) while maintaining or improving language modeling performance, supported by ablations that confirm the importance of alignment, the causal-estimation step, and toxicity screening for data quality. This approach offers a practical path to debiasing lightweight PLMs by harnessing LLM-derived knowledge with alignment-aware transfer, enabling safer and fairer NLP deployments across downstream tasks.

Abstract

Pre-trained language models (PLMs) are trained on data that inherently contains gender biases, leading to undesirable impacts. Traditional debiasing methods often rely on external corpora, which may lack quality, diversity, or demographic balance, affecting the effectiveness of debiasing. With the rise of large language models and their extensive knowledge, we propose enhancing fairness (Fair-Gender) in PLMs by absorbing coherent, attribute-balanced, and semantically rich sentences. However, these sentences cannot be directly used for debiasing due to alignment issues and the risk of negative transfer. We address this by applying causal analysis to estimate causal effects, filtering out unaligned sentences, and identifying aligned ones for incorporation into PLMs, thereby ensuring positive transfer. Experiments show that our approach significantly reduces gender biases in PLMs while preserving their language expressiveness.

Figures (5)

• Figure 1: Our comprehensive debiasing framework Fair-Gender.
• Figure 2: The comparison of structural causal model between conventional methods and our Fair-Gender.
• Figure 3: Toxicity detection of sentences, and ablation versions.
• Figure 4: The number impact of the generated pairwise sentences.
• Figure 5: $t$-SNE plots on BERT. Red: female, blue: male, and orange: neutral target words.