Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models

TL;DR

The paper argues that relying on Outcome Accuracy alone leads to deceptive alignment in reward modeling and LLM-based judgment. It introduces Rationale Consistency and MetaJudge to quantify how well a model's reasoning aligns with human judgment, revealing substantial gaps not captured by accuracy alone. By training Generative Reward Models with a hybrid objective that rewards both correct outcomes and human-aligned rationales, the approach achieves state-of-the-art results on RM-Bench and JudgeBench and improves RLHF performance, while mitigating rationale degeneration. The work highlights the importance of explicit rationale supervision for robust, human-aligned evaluation and discusses limitations related to annotation scalability and the need for robust human-AI collaboration in future work.

Abstract

Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training.

Figures (14)

• Figure 1: Outcome Accuracy vs. Human Rationale Consistency. Rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment.
• Figure 2: (a) Insensitivity to the Evaluator Model: Rationale consistency computed by different evaluators are highly correlated ($R^2=0.983$). (b) Generalization Across Domains and Annotators: Model rankings remain largely consistent between HelpSteer3-Atomic and CW-Atomic (Spearman $\rho=0.85$).
• Figure 3: While outcome accuracy remains comparable across methods, the absence of rationale supervision causes a significant collapse in reasoning quality.
• Figure 4: Degeneration and recovery of reasoning. Outcome-only training degrades rationales into superficial shortcuts: Criterion-Grounded (CG) and Generic/Style (GS). In contrast, our method successfully restores Evidence-Grounded (EG) reasoning to 98.7%.
• Figure 5: Item-level rationale flaw rates for three settings. Outcome-only training amplifies these flaws, while adding rationale supervision reduces them across the board.