Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization

TL;DR

This work investigates how to reliably evaluate reward models (RMs) used in RLHF by reframing evaluation through reward overoptimization—the phenomenon where optimizing for a learned RM's reward degrades alignment with true human preferences and downstream performance. It introduces a quantifiable degree of overoptimization, $\gamma$, derived from BoN-based learning curves, and demonstrates that RM evaluation designs must emphasize distributional differences between chosen and rejected responses, response diversity, and multi-pairwise comparisons. The study shows that while $\gamma$ correlates with downstream performance, an excessively high correlation can degrade this link due to Goodhart's law; hence, $\gamma$ should guide benchmarking rather than be the objective. The authors validate their design across mathematics, code, and safety domains, suggesting practical benchmarks that better predict RM-guided policy performance and highlighting the importance of cross-model diversity and robust evaluation metrics for RLHF systems.

Abstract

Reward models (RMs) play a crucial role in reinforcement learning from human feedback (RLHF), aligning model behavior with human preferences. However, existing benchmarks for reward models show a weak correlation with the performance of optimized policies, suggesting that they fail to accurately assess the true capabilities of RMs. To bridge this gap, we explore several evaluation designs through the lens of reward overoptimization\textemdash a phenomenon that captures both how well the reward model aligns with human preferences and the dynamics of the learning signal it provides to the policy. The results highlight three key findings on how to construct a reliable benchmark: (i) it is important to minimize differences between chosen and rejected responses beyond correctness, (ii) evaluating reward models requires multiple comparisons across a wide range of chosen and rejected responses, and (iii) given that reward models encounter responses with diverse representations, responses should be sourced from a variety of models. However, we also observe that a extremely high correlation with degree of overoptimization leads to comparatively lower correlation with certain downstream performance. Thus, when designing a benchmark, it is desirable to use the degree of overoptimization as a useful tool, rather than the end goal.

Figures (15)

• Figure 1: Relationship between existing benchmark scores for reward models ($x$-axis) and downstream performance ($y$-axis) for BoN sampling and PPO. Each dashed line indicates a best-fit trend with its $r^2$ value, demonstrating that existing benchmarks exhibit low $r^2$. Relying on such unreliable benchmarks to develop reward models is likely to hinder progress in RLHF.
• Figure 2: An example of reward overoptimization. The KL divergence represents the degree of optimization.
• Figure 3: The results of the degree of overoptimization ($\gamma_{\text{oracle}}$) under oracle reward setting. Higher values of $\gamma$ indicate a greater tendency for reward to degrade as optimized continues. The value of $\gamma$ for each RMs are reported in Table \ref{['tab:app_rm_gamma']}.
• Figure 4: Relationship between degree of overoptimization $\gamma_{\text{oracle}}$ and downstream performance (BoN and PPO) using MetaMATH-Mistral-7B as policy model. The results demonstrate that $\gamma_\text{oracle}$ strongly correlates with the downstream performance. Results for $\gamma_{\text{gold}}$ and Llama3-8B-Instruct are provided in Figure \ref{['fig:app_roo_down']}.
• Figure 5: The correlation ($r^2$) between evaluation results across different designs and downstream performance with two policy models. (Left) Results for single pairwise evaluation designs. (Right) Results for multi-pairwise evaluation designs, demonstrating stronger correlation compare to single pairwise comparisons.