Reward Model Perspectives: Whose Opinions Do Reward Models Reward?

TL;DR

The paper investigates whose opinions reward models (RMPs) reward by auditing reward models for sociodemographic biases and steerability. It formalizes RM perspectives through distributions of RM-derived opinions and an alignment metric that compares these distributions to demographic respondent distributions, using distances such as Jensen–Shannon and Wasserstein within a Bradley–Terry RM framework. By evaluating seven open-source RMs on datasets BBQ, OpinionQA, PRISM, and StereoSet, it finds that absolute alignment is model-dependent, but relative alignment across demographic groups is robust, with detectable stereotypes varying by model and dataset. It also shows that prompt-based in-context steering toward a target group yields little, often statistically insignificant, improvements, underscoring the need for bias-aware preference learning and beyond-prompt mitigation for AI alignment.

Abstract

Reward models (RMs) are central to the alignment of language models (LMs). An RM often serves as a proxy for human preferences to guide downstream LM behavior. However, our understanding of RM behavior is limited. Our work (i) formalizes a framework for measuring the alignment of opinions captured by RMs, (ii) investigates the extent to which RMs demonstrate sociodemographic biases, and (iii) explores the effects of prompting to steer rewards towards the preferences of a target group. We study the subjective and diverse perspectives on controversial topics, which allows us to quantify RM perspectives in terms of their opinions, attitudes, and values. We show that RMs are poorly aligned with several demographic groups and can systematically reward harmful stereotypes, and steering alone is not enough to overcome these limitations. Our findings underscore the need for more careful consideration of RM behavior in model alignment during preference learning to prevent the propagation of unwanted social biases in the language technologies that we use.

Figures (20)

• Figure 1: The average ranks of demographic alignment in OpinionQA. We plot the average rank ($\downarrow$ better aligned) across all RMs for every demographic group. Certain sociodemographic groups, such as identifying with the political party of "Other" or having an income of less than $30,000, received systematically better rankings across RMs than individuals in certain religious groups or groups with more extreme political ideologies.
• Figure 2: Ranks ($\downarrow$) of rewards by demographic group on OpinionQA. We showcase the alignment metric per RM (bar), the average ranking across all RMs per demographic group (top), and the detailed ranks per RM per demographic group (panel). Demographic groups that are better represented receive lower ranks (darker circles) and higher alignment values (larger circles) than groups that are poorly represented. The absolute alignment (size) appears to be model dependent. The relative alignment (hue) is fairly consistent between different demographic groups across RMs, meaning every demographic group obtains a similar rank across all models.
• Figure 3: Alignment ($\uparrow$) with PRISM respondents. Absolute alignment is dependent on the choice of the RM (color), although relative alignment within an RM remains sensitive to the demographic group (shape).
• Figure 4: Confusion matrix of RM predicted labels on BBQ. The heatmap shows the number of samples that have a predicted label of Stereotyped (S), Unknown (?), and Unstereotyped (U) against the expected gold label.
• Figure 5: Examples of RQ2 data. The BBQ data contains both an ambiguous and a disambiguous scenario via the optional context in the brackets ([CONTEXT]).