reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs

TL;DR

This work shows that state-of-the-art reward models are brittle to meaning- and ranking-preserving input transformations, revealing overfitting to standard benchmarks. It introduces reWordBench to systematically probe RM robustness through controlled, naturalistic, and domain-targeted transformations and proposes a paraphrase-based regularization method to enforce score consistency across equivalent inputs. The regularized RMs exhibit improved robustness on reWordBench and deliver higher-quality outputs in downstream alignment, including substantial gains against standardly trained RMs. The findings highlight the importance of robustness-aware training for reliable evaluation and alignment, with practical implications for safer and more effective AI systems.

Abstract

Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time algorithms. However, while recent reward models increase performance on standard benchmarks, this may partly be due to overfitting effects, which would confound an understanding of their true capability. In this work, we scrutinize the robustness of reward models and the extent of such overfitting. We build **reWordBench**, which systematically transforms reward model inputs in meaning- or ranking-preserving ways. We show that state-of-the-art reward models suffer from substantial performance degradation even with minor input transformations, sometimes dropping to significantly below-random accuracy, suggesting brittleness. To improve reward model robustness, we propose to explicitly train them to assign similar scores to paraphrases, and find that this approach also improves robustness to other distinct kinds of transformations. For example, our robust reward model reduces such degradation by roughly half for the Chat Hard subset in RewardBench. Furthermore, when used in alignment, our robust reward models demonstrate better utility and lead to higher-quality outputs, winning in up to 59% of instances against a standardly trained RM.

Figures (10)

• Figure 1: A state-of-the-art RM on RewardBench (Skywork/Skywork-Reward-Gemma-2-27B-v0.2) drastically changes its assigned rewards and flips its preference when only a few (bolded) words in the input change. Explicitly regularizing RMs during training (§\ref{['sec:training']}) improves its robustness and maintains the preference. The rewards are normalized into $[0, 1]$. The example is taken from RewardBench and originally from zeng2024evaluating.
• Figure 2: The ranking accuracy of reward models under meaning- or ranking-preserving transformations. SOTA RMs consistently suffer from performance degradation when inputs are slightly transformed. Full results broken down by specific transformations are in §\ref{['sec:full-results']}.
• Figure 3: Ranking accuracy drop of our regularized reward models under the Ignore Above transformation (§\ref{['sec:controlled-transformations']}), with different regularization strengths $\alpha$. The $x$-axis is not linear with respect to $\alpha$. The RM robustness improves with increasing regularization strength.
• Figure 4: Comparing outputs aligned by our regularized RM vs. a standard-trained RM (and the unaligned SFT model). We show how often (%) each model wins according to an LM judge, or when they produce identical outputs (tie). Our regularized RM consistently leads to better outputs in alignment compared to a standard-trained RM.
• Figure 5: The change of RM ranking accuracy under meaning- or ranking-preserving (controlled) reWordBench transformations.