The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

TL;DR

This paper investigates whether alignment of large language models (LLMs) to human preferences can preserve or amplify gender-diverse biases, focusing on transgender, nonbinary, and other TGNB identities. It evaluates 16 models across pretraining, supervised fine-tuning (SFT), and Direct Preference Optimization (DPO), analyzing outputs on TGNB-disclosure prompts, preference-data sources, and implicit reward signals. The study finds that DPO-aligned models can amplify TGNB harms—specifically stigmatization and gender non-affirmative language—when initialized from certain reference models, and that reward signals can encode and transfer TGNB bias. It proposes a flexible reward-signal analysis framework and advocates community-informed bias evaluation frameworks, standardized reward-signal assessment, and greater transparency in alignment pipelines to mitigate TGNB harms in LLMs. The work highlights the need for inclusive, practice-aware evaluation and intervention strategies to ensure aligned LLMs serve TGNB communities without reinforcing real-world harms.

Abstract

Natural-language assistants are designed to provide users with helpful responses while avoiding harmful outputs, largely achieved through alignment to human preferences. Yet there is limited understanding of whether alignment techniques may inadvertently perpetuate or even amplify harmful biases inherited from their pre-aligned base models. This issue is compounded by the choice of bias evaluation benchmarks in popular preference-finetuned models, which predominantly focus on dominant social categories, such as binary gender, thereby limiting insights into biases affecting underrepresented groups. Towards addressing this gap, we center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias in LLMs. Our key contributions include: 1) a comprehensive survey of bias evaluation modalities across leading preference-finetuned LLMs, highlighting critical gaps in gender-diverse representation, 2) systematic evaluation of gender-diverse biases across 16 models spanning Direct Preference Optimization (DPO) stages, uncovering harms popular bias benchmarks fail to detect, and 3) a flexible framework for measuring harmful biases in implicit reward signals applicable to other social contexts. Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning (SFT), and can amplify two forms of real-world gender-diverse harms from their base models: stigmatization and gender non-affirmative language. We conclude with recommendations tailored to DPO and broader alignment practices, advocating for the adoption of community-informed bias evaluation frameworks to more effectively identify and address underrepresented harms in LLMs.

Figures (7)

• Figure 1: Bias benchmarks employed by top 15 performing preference-tuned LLMs reported by Chatbot Arena Leaderboard chiang2024chatbot across socially-relevant categories. Evaluations fully cover binary gender bias, with limited evaluation for gender-diverse minorities and other socially-salient dimensions.
• Figure 2: % of texts labeled as negative regard across gender groups, textual disclosure forms, and model alignment stages.
• Figure 3: Example negative regard amplification after DPO, prompt is bold.
• Figure 4: Reward extraction procedure for WinoQueer ex: "Transgender people are unfit for professional work". We simulate chosen vs. rejected assistant responses to extract reward signals from DPO-aligned LLMs.
• Figure 5: LLMs reflect at least moderate agreement (>0.4) across alignment stages and model families.