Disclosure and Mitigation of Gender Bias in LLMs

TL;DR

This work tackles gender bias in Large Language Models by introducing an indirect probing framework based on conditional generation to reveal explicit and implicit bias without relying on explicit gender terms. It compares three probing strategies—Naturally-sourced, LLM-generated, and Template-based inputs—and defines three metrics, GAS for explicit bias and GLD/ADD for implicit bias. Across ten LLMs, the study finds pervasive bias and shows that larger or aligned models can amplify explicit bias while sometimes reducing implicit bias; it then demonstrates three mitigation approaches—Hyperparameter Tuning, Instruction Guiding, and Debias Tuning—with Debias Tuning delivering the strongest, transferable improvements across metrics and datasets. The results highlight the potential and practical impact of indirect bias assessment and targeted debiasing, while discussing limitations and ethical considerations for deploying such techniques in real-world systems, and also providing a foundation for future data-informed debiasing strategies. A key contribution is the Debias Tuning framework, formalized through the losses $L_d$, $L_g$, and $L_l$ combined as $L = L_d + L_g + L_l$, enabling effective bias reduction via QLoRA.

Abstract

Large Language Models (LLMs) can generate biased responses. Yet previous direct probing techniques contain either gender mentions or predefined gender stereotypes, which are challenging to comprehensively collect. Hence, we propose an indirect probing framework based on conditional generation. This approach aims to induce LLMs to disclose their gender bias even without explicit gender or stereotype mentions. We explore three distinct strategies to disclose explicit and implicit gender bias in LLMs. Our experiments demonstrate that all tested LLMs exhibit explicit and/or implicit gender bias, even when gender stereotypes are not present in the inputs. In addition, an increased model size or model alignment amplifies bias in most cases. Furthermore, we investigate three methods to mitigate bias in LLMs via Hyperparameter Tuning, Instruction Guiding, and Debias Tuning. Remarkably, these methods prove effective even in the absence of explicit genders or stereotypes.

Figures (7)

• Figure 1: Explicit and Implicit Gender Bias in LLMs.
• Figure 2: Gender Attribute Score (GAS). Naturally-sourced is the combination of Naturally-sourced (STS-B) and (SNLI), while LLM-generated is the combination of LLM-generated (1) and (2).
• Figure 3: Gender Attribute Score (GAS) for subsets.
• Figure 4: Gender Logits Difference (GLD).
• Figure 5: Attribute Distribution Distance (ADD).