Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance

TL;DR

The paper tackles inductive biases in reward models used for RLHF, which can drive reward hacking and poor generalization. It introduces Debiasing via Information optimization for RMs (DIR), an information-theoretic framework that maximizes $I(1_{y\succ\bar{y}}; x,y,\bar{y})$ while minimizing $I(1_{y\succ\bar{y}}; b)$ through variational bounds (Barber-Agakov and CLUB) and a relative-bias regularizer. DIR handles non-linear, complex biases such as length, sycophancy, and format, and demonstrates improved RM reliability and RLHF performance across multiple benchmarks (GSM8K, MMLU, ArenaHard, MT-Bench, RM-Bench) with tunable debiasing strength. The work provides theoretical grounding and practical results, and includes code and training recipes to facilitate adoption."

Abstract

Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large language models (LLMs) with human values. However, RM training data is commonly recognized as low-quality, containing inductive biases that can easily lead to overfitting and reward hacking. For example, more detailed and comprehensive responses are usually human-preferred but with more words, leading response length to become one of the inevitable inductive biases. A limited number of prior RM debiasing approaches either target a single specific type of bias or model the problem with only simple linear correlations, \textit{e.g.}, Pearson coefficients. To mitigate more complex and diverse inductive biases in reward modeling, we introduce a novel information-theoretic debiasing method called \textbf{D}ebiasing via \textbf{I}nformation optimization for \textbf{R}M (DIR). Inspired by the information bottleneck (IB), we maximize the mutual information (MI) between RM scores and human preference pairs, while minimizing the MI between RM outputs and biased attributes of preference inputs. With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations, broadly extending the real-world application scenarios for RM debiasing methods. In experiments, we verify the effectiveness of DIR with three types of inductive biases: \textit{response length}, \textit{sycophancy}, and \textit{format}. We discover that DIR not only effectively mitigates target inductive biases but also enhances RLHF performance across diverse benchmarks, yielding better generalization abilities. The code and training recipes are available at https://github.com/Qwen-Applications/DIR.

Figures (6)

• Figure 1: The proposed DIR framework. The architecture of the reward model is considered as a backbone transformer and an RM score head. The original RM ranking loss ${\mathcal{L}}_\text{Preference}(\phi)$ is calculated based on the outputs of the score head between each preference pair. The last hidden states $({\bm{h}}^w, {\bm{h}}^l)$ of the transformer backbone are collected as the representation ${\bm{H}}$. The debiasing loss ${\mathcal{L}}_\text{debiasing}(\phi,\psi)$ is computed between the inductive bias label ${\bm{b}}$ and the output of the debiasing head with parameter $\psi$.
• Figure 2: Evaluation of length bias in RMs on RM-Bench. We calculated the correlation between response length and reward score for RMs trained with different methods. Our approach yields the lowest Pearson correlation coefficient ($r=0.468$), proving its effective ability in assigning more uniform reward scores.
• Figure 3: RLHF Evaluation on ArenaHard-v0.1 with different length-debiased RMs. (a) Head-to-head win rates. Policies are PPO fine-tuned from specified base models (from left to right: OpenLlama3-8B-SFT, Llama3.1-8B-Instruct, and Llama3.1-8B-Instruct, respectively) using five different RMs, which then act as challengers against opponents. (b) Average response length comparison.
• Figure 4: PPO training dynamics across key metrics. Our RM obtains a higher policy score and demonstrates better training stability.
• Figure 5: Ablation study on the debiasing coefficient $\lambda$. The plot shows the trade-off between preference learning performance (RewardBench Accuracy, blue) and the bias metric (e.g., Pearson $r$, green). $\lambda=1$ achieves the best balance.