Detecting Prefix Bias in LLM-based Reward Models

TL;DR

The paper demonstrates that LLM-based reward models trained with RLHF exhibit prefix bias, where short identity prefixes subtly steer model preferences and accuracy. It introduces two metrics, auto-influence and cross-influence, to quantify this bias across gender and race prefixes on multiple datasets and architectures, revealing that bias largely arises from training data rather than model architecture. A data-augmentation strategy is proposed and shown to significantly reduce both auto- and cross-influence with minimal impact on baseline accuracy, suggesting a practical path to fairer reward models. The work highlights the need for bias-aware dataset design and evaluation in RLHF pipelines and provides generalizable methods for auditing reward models against prefix- or context-based biases.

Abstract

Reinforcement Learning with Human Feedback (RLHF) has emerged as a key paradigm for task-specific fine-tuning of language models using human preference data. While numerous publicly available preference datasets provide pairwise comparisons of responses, the potential for biases in the resulting reward models remains underexplored. In this work, we introduce novel methods to detect and evaluate prefix bias -- a systematic shift in model preferences triggered by minor variations in query prefixes -- in LLM-based reward models trained on such datasets. We leverage these metrics to reveal significant biases in preference models across racial and gender dimensions. Our comprehensive evaluation spans diverse open-source preference datasets and reward model architectures, demonstrating susceptibility to this kind of bias regardless of the underlying model architecture. Furthermore, we propose a data augmentation strategy to mitigate these biases, showing its effectiveness in reducing the impact of prefix bias. Our findings highlight the critical need for bias-aware dataset design and evaluation in developing fair and reliable reward models, contributing to the broader discourse on fairness in AI.

Figures (12)

• Figure 1: (Left) Average winrate deviation (auto-influence) and (Right) average accuracy deviation (cross-influence) for different dataset-model combinations, using the gender group prefixes.
• Figure 2: Distribution of pairwise winrates for SHP datasets. Each bar represents the distribution of winrates across all model architectures (excluding opt-350m). Positive values mean the first group is preferred over the second group in the comparison. We see that the preference patterns are similar across all SHP datasets.
• Figure 3: Distribution of pairwise winrates for anthropic datasets. Each bar represents the distribution of winrates across all model architectures (excluding opt-350m). Positive values mean the first group is preferred over the second group in the comparison.
• Figure 6: Augmented training results for the gender prefixes. "aug" suffix refers to the augmented model trained on corresponding data, "aug-gen" refers to a model trained on augmented data for different prefixes, a lack of suffix indicates the reward model trained on raw data, "-0shot" suffix denotes the base LLM used with zero-shot prompting.
• Figure 7: Augmented training results for the race prefixes. "aug" suffix refers to the augmented model trained on corresponding data, "aug-gen" refers to model trained on augmented data for different prefixes, no suffix refers to the reward model trained on raw data, "-0shot" suffix denotes the base LLM used with zero-shot prompting.