R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging

TL;DR

The paper identifies Spurious Correctness as a core failure mode where GenRMs justify correct preferences with flawed reasoning, showing that label accuracy alone poorly predicts downstream RLHF performance. It introduces a rationale-aware benchmarking framework with a MetaRM and defines metrics like S-Corr and F-Score to quantify reasoning misalignment. To address this, it proposes Rationale-Centric Alignment (R-Align), which uses Meta-Judging to reward not just correct labels but correct, aligned reasoning, and demonstrates reductions in S-Corr and improved RLHF outcomes. Across diverse domains, R-Align yields more robust actor performance and stronger correlations between reasoning fidelity and downstream success, highlighting the practical importance of aligning reasoning processes in GenRMs for reliable RLHF.

Abstract

Reinforcement Learning from Human Feedback (RLHF) remains indispensable for aligning large language models (LLMs) in subjective domains. To enhance robustness, recent work shifts toward Generative Reward Models (GenRMs) that generate rationales before predicting preferences. Yet in GenRM training and evaluation, practice remains outcome-label-only, leaving reasoning quality unchecked. We show that reasoning fidelity-the consistency between a GenRM's preference decision and reference decision rationales-is highly predictive of downstream RLHF outcomes, beyond standard label accuracy. Specifically, we repurpose existing reward-model benchmarks to compute Spurious Correctness (S-Corr)-the fraction of label-correct decisions with rationales misaligned with golden judgments. Our empirical evaluation reveals substantial S-Corr even for competitive GenRMs, and higher S-Corr is associated with policy degeneration under optimization. To improve fidelity, we propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment. R-Align reduces S-Corr on RM benchmarks and yields consistent gains in actor performance across STEM, coding, instruction following, and general tasks.

Figures (8)

• Figure 1: An illustration of "Spurious Correctness". The GenRM correctly prefers Response A over Response B, but generates a flawed rationale. While the Golden Judgment captures the true content difference (empathy vs. insensitivity), the GenRM relies solely on the superficial feature of bulleted list formatting.
• Figure 2: Divergent RLHF outcomes despite comparable GenRM benchmark accuracy. Left: reward curves during RL training. Right: periodic downstream evaluation shows continued improvement with Qwen3-14B but degradation with RRM-32B.
• Figure 3: Overview of the MetaRM framework. (a) GenRM RLVR (Baseline): The model is optimized solely on outcome correctness, receiving rewards ($R=1$) for accurate preference labels regardless of the reasoning quality. (b) Rationale-Centric Alignment (Ours): Incorporates a MetaRM to enforce process supervision; rewards are granted only when both the label is correct and the rationale is logically consistent with the golden reference, effectively penalizing spurious correctness. (c) F-Score vs. L-Acc: Visualizes F-Score as a strict subset of L-Acc, filtering out spurious correctness where the model is right for the wrong reasons.
• Figure 4: Benchmark results on HelpSteer3, RewardBench2, and PPE-Preference. The numerical labels on top of the bars denote the Label Accuracy. The solid bars represent the F-Score, while the hatched areas indicate the proportion of Spurious Correctness.
• Figure 5: Correlation analysis between benchmark metrics on HelpSteer3 and downstream RLHF performance. Left: Label Accuracy (L-Acc). Right: Fidelity Score (F-Score).