Reward Models Inherit Value Biases from Pretraining

TL;DR

The paper demonstrates that reward models implicitly inherit value biases from their pretraining base models, with Llama-based RMs showing agency bias and Gemma-based RMs showing communion bias across multiple value axes. By combining exhaustive token search with psycholinguistic corpora (Big Two and MFD2) and introducing implicit reward scores via log-probability differences, the authors link downstream RM behavior to base-model log probabilities and to an explicit implicit reward framework MWLR: $MWLR = \tfrac{1}{2}(p_1 + p_2) \cdot (\log p_2 - \log p_1)$. They further show that these biases originate in pretraining and persist through RM training across data sources and scales, though increasing preference data modestly mitigates the gap. The results highlight that alignment efforts must consider pretraining choices and base-model families, as safety and value alignment are deeply shaped at the pretraining stage and not fully washable via fine-tuning alone.

Abstract

Reward models (RMs) are central to aligning large language models (LLMs) with human values but have received less attention than pre-trained and post-trained LLMs themselves. Because RMs are initialized from LLMs, they inherit representations that shape their behavior, but the nature and extent of this influence remain understudied. In a comprehensive study of 10 leading open-weight RMs using validated psycholinguistic corpora, we show that RMs exhibit significant differences along multiple dimensions of human value as a function of their base model. Using the "Big Two" psychological axes, we show a robust preference of Llama RMs for "agency" and a corresponding robust preference of Gemma RMs for "communion." This phenomenon holds even when the preference data and finetuning process are identical, and we trace it back to the logits of the respective instruction-tuned and pre-trained models. These log-probability differences themselves can be formulated as an implicit RM; we derive usable implicit reward scores and show that they exhibit the very same agency/communion difference. We run experiments training RMs with ablations for preference data source and quantity, which demonstrate that this effect is not only repeatable but surprisingly durable. Despite RMs being designed to represent human preferences, our evidence shows that their outputs are influenced by the pretrained LLMs on which they are based. This work underscores the importance of safety and alignment efforts at the pretraining stage, and makes clear that open-source developers' choice of base model is as much a consideration of values as of performance.

Figures (13)

• Figure 1: Value preferences (token ranks) from 10 leading RewardBench RMs based on Gemma and Llama for words related to different moral concepts. (a) Preferences for the Big-Two dimensions, for positively-framed prompts (top) and negatively-framed prompts (bottom). (b) Same as (a), for 5 MFD2 dimensions. Dots show mean ± s.e. of the median ranking of each single model, averaged over prompts; black markers indicate grand mean ± s.e.; violin plots visualize the density of the distribution. Stars: $p < .0001$ (Bonferroni-corrected permutation $t$-tests).
• Figure 2: Log probabilities in both the instruction-tuned and pre-trained versions of the Gemma and Llama base models reveal the same agency/communion split observed in their respective RMs' reward scores. Violin plots show the median rank of the Big-Two nouns according to the log probabilities assigned by the (a) instruction-tuned and (b) pre-trained versions of Gemma 2 2B and Llama 3.2 3B. Each dot corresponds to one of our positively (top) or negatively (bottom) valenced prompts. *** $p<0.001$, ** $p<0.01$, FDR-corrected. Boxes show median (white line) and interquartile ranges and whiskers extend to the ends of the distribution excluding outliers.
• Figure 3: MWLR scores for "Love" and "Freedom" (averaged over all variants of whitespace and capitalization) for the "greatest thing ever" prompt across all Gemma 2 (2--27B) and Llama 3 (1--70B) models reveal a gap in all 21 combinations, which increases with model size.
• Figure 4: (a) A pair of Llama and Gemma RMs trained using Skywork 80k preference data, checkpointed every 1000 steps during training, evaluated with the prompt "What, in one word, is the greatest thing ever?" (b) Ablation studies for data source (Skywork $\triangle$ vs. Unified Feedback $\circ$) and quantity (13k, 53k, 80k and 106k), depicting final checkpoints of all runs. We show the gap in preference over the Big Two between Llama (blue) and Gemma (red) at the end of training. For comparability, we also include data from Gemma- and Llama-based "GRMs" trained by yang2024regularizing using a combination of regularized BT on a 632k mixture of open-source datasets ($\diamond$) plus standard BT on Skywork.
• Figure 5: Differences in preferred tokens during the early and final stages of training. Each figure shows the top and bottom five tokens from the Big Two corpus that most dramatically changed in their ranked preferences between our earliest checkpoint (step 1000) and our latest checkpoint (step 9578). Through training, the Gemma RM increases its scores for "agency" tokens, while the Llama RM increases its scores for "communion" tokens.