Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions

TL;DR

Reward models often generalize poorly to unseen prompts, a shortcoming not captured by static benchmarks. The authors introduce Pairwise Maximum Discrepancy Competition (PMDC), a dynamic evaluation framework that actively samples prompt–response pairs from a large open-domain pool to maximize disagreement between RMs, adjudicates them with an LLM-based oracle in a 2AFC setup, and aggregates results with a Bradley–Terry model to produce global rankings and win-rate landscapes. On 10 representative RMs, PMDC reveals substantial rank reshuffling relative to traditional benchmarks and provides diagnostics of systematic generalization failures, illustrating the framework’s diagnostic value. Beyond evaluation, PMDC-identified samples can be used for targeted fine-tuning, improving RM robustness and alignment fidelity by focusing on high-disagreement cases. The work offers a scalable, annotation-efficient method for probing RM generalization and guiding practical improvements in reward modeling.

Abstract

Reward models (RMs) are central to aligning large language models, yet their practical effectiveness hinges on generalization to unseen prompts and shifting distributions. Most existing RM evaluations rely on static, pre-annotated preference datasets, which provide limited coverage and often fail to faithfully assess generalization in open-world settings. We introduce Pairwise Maximum Discrepancy Competition (PMDC), a dynamic and annotation-efficient framework for evaluating RM generalization using a large, unlabeled, open-domain prompt pool. PMDC actively selects prompt--response pairs that maximize disagreement between two RMs, yielding a compact set of highly contentious test cases. These cases are adjudicated by an oracle, and the resulting outcomes are aggregated via a Bradley--Terry model to produce a global ranking and pairwise win-rate landscape of RMs. We apply PMDC to re-evaluate 10 representative RMs and observe substantial rank reshuffling compared with conventional benchmarks. Qualitative analyses further uncover systematic generalization failures, providing valuable insights for improving reward modeling.

Figures (6)

• Figure 1: Overview of the proposed PMDC framework. (a) Data generation and scoring: Sample prompts and LLMs to build prompt-response pairs, which are then scored by reward models (RMs). (b) Sample selection: Based on the Maximum Discrepancy Competition principle, select top-$k$ pairs (with maximum RM preference discrepancy) to form an evaluation subset. (c) Comparison & ranking: Annotate the selected QA pairs with an Oracle (i.e., LLM-based Judge) to rank responses, compare Oracle results with RMs to build a win-rate matrix, and convert the pairwise comparisons into RMs’ global ranking using the Bradley-Terry model.
• Figure 2: Pairwise win-rate heatmap across RMs on Maximum Discrepancy samples.
• Figure 3: Rank comparison between PMDC and RewardBench2. Horizontal lines connect each reward model’s ranking under RewardBench2 (blue) and PMDC (red).
• Figure 4: Spearman correlation of PMDC ranks across top-$k$ values. The dashed red line at 0.95 highlights high rank consistency.
• Figure 5: PMDC's rank across five independent runs. The heatmap shows the rank of each RM in each run, with rank values annotated in individual cells.