CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria

TL;DR

This work addresses the gap between reward-model benchmarks and real RL effectiveness by introducing CE-RM-4B, a pointwise generative reward model trained with a two-stage rollout to unify evaluation criteria based on the query. It leverages a compact, carefully filtered dataset (~5.7K instances) and a training regime that explicitly optimizes criteria generation and evaluation generation, achieving strong performance on common RM benchmarks, especially in Best-of-N settings. The method demonstrates improved robustness and practical RL improvements, outperforming larger, traditional pairwise GRMs with substantially less data and compute. The approach contributes a practical framework for reward modeling that better aligns with RLHF objectives and real-world evaluation demands, with potential extensions to tool use and broader RL contexts.

Abstract

Automatic evaluation is crucial yet challenging for open-ended natural language generation, especially when rule-based metrics are infeasible. Compared with traditional methods, the recent LLM-as-a-Judge paradigms enable better and more flexible evaluation, and show promise as generative reward models for reinforcement learning. However, prior work has revealed a notable gap between their seemingly impressive benchmark performance and actual effectiveness in RL practice. We attribute this issue to some limitations in existing studies, including the dominance of pairwise evaluation and inadequate optimization of evaluation criteria. Therefore, we propose CE-RM-4B, a pointwise generative reward model trained with a dedicated two-stage rollout method, and adopting unified query-based criteria. Using only about 5.7K high-quality data curated from the open-source preference dataset, our CE-RM-4B achieves superior performance on diverse reward model benchmarks, especially in Best-of-N scenarios, and delivers more effective improvements in downstream RL practice.

Figures (6)

• Figure 1: The variance of the evaluation scores corresponding to the same criteria trajectory (in-criteria), and to all criteria trajectories (all-criteria) for each instance, as well as the criteria rewards during RL.
• Figure 2: The results of the ablation experiments that compare our CE-RM-4B with other reward models trained without different key components of our methods.
• Figure 3: During training, the evolution of performance of the policy model across three evaluation scenarios, along with the averaged evaluation scores that the reward model assigns to the policy rollouts. In the RL practice presented here, the policy model is Qwen3-8B and the reward model is our CE-RM-4B (without test-time scaling), with GRPO and a group size of 8.
• Figure 4: The prompt for the first evaluation setting and Eq. (\ref{['eq1']}) in Section \ref{['preliminary']}.
• Figure 5: The prompt for the second evaluation setting and Eq. (\ref{['eq2']}) in Section \ref{['preliminary']}.