Rejected Dialects: Biases Against African American Language in Reward Models

TL;DR

This work investigates anti-AAL biases in reward models used for alignment of large language models. It introduces a framework combining machine-translated (RB-WME/RB-AAL) and human-translated (DG) dialect data to probe 17 reward-models under RLHF and DPO. The findings show that reward models are less accurate with AAL inputs, tend to disprefer AAL content, and steer conversations toward White Mainstream English, revealing representational harms and ethical concerns. The study highlights the need for dialect-aware evaluation, broader AAL representation in preference data, and Value-Sensitive Design that actively involves AAL communities in AI system development.

Abstract

Preference alignment via reward models helps build safe, helpful, and reliable large language models (LLMs). However, subjectivity in preference judgments and the lack of representative sampling in preference data collection can introduce new biases, hindering reward models' fairness and equity. In this work, we introduce a framework for evaluating dialect biases in reward models and conduct a case study on biases against African American Language (AAL) through several experiments comparing reward model preferences and behavior on paired White Mainstream English (WME) and both machine-translated and human-written AAL corpora. We show that reward models are less aligned with human preferences when processing AAL texts vs. WME ones (-4\% accuracy on average), frequently disprefer AAL-aligned texts vs. WME-aligned ones, and steer conversations toward WME, even when prompted with AAL texts. Our findings provide a targeted analysis of anti-AAL biases at a relatively understudied stage in LLM development, highlighting representational harms and ethical questions about the desired behavior of LLMs concerning AAL.

Figures (1)

• Figure 1: We analyze reward model scores for White Mainstream English (W) and African American Language (A) texts across various prompt-continuation settings. Vertical dotted lines indicate machine translations and checkmarks/Xs indicate human preferences between alternatives. Our findings point to representational and quality-of-service harms for AAL speakers.