OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

TL;DR

OpenRM presents a tool-augmented reward model trained with GRPO to evaluate long-form, knowledge-intensive outputs by actively retrieving external evidence. The method integrates a composite reward that balances intermediate tool-use decisions with final answer accuracy, and uses controllable data synthesis to generate scalable training pairs. Empirical results show strong in-domain and out-of-domain performance, better generalization with limited data, and positive impacts on downstream alignment when used for data selection and inference-time decision-making. This work demonstrates that grounding evaluations in verifiable external information can significantly improve the reliability and scalability of reward-based alignment for LLMs.

Abstract

Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and long-form tasks, where evaluating correctness requires grounding beyond the model's internal knowledge. This limitation hinders them from reliably discriminating subtle quality differences, especially when external evidence is necessary. To address this, we introduce OpenRM, a tool-augmented long-form reward model that systematically judges open-ended responses by invoking external tools to gather relevant evidence. We train OpenRM with Group Relative Policy Optimization (GRPO) on over 27K synthesized pairwise examples generated through a controllable data synthesis framework. The training objective jointly supervises intermediate tool usage and final outcome accuracy, incentivizing our reward model to learn effective evidence-based judgment strategies. Extensive experiments on three newly-collected datasets and two widely-used benchmarks demonstrate that OpenRM substantially outperforms existing reward modeling approaches. As a further step, we integrate OpenRM into both inference-time response selection and training-time data selection. This yields consistent gains in downstream LLM alignment tasks, highlighting the potential of tool-augmented reward models for scaling reliable long-form evaluation.

Figures (3)

• Figure 1: Illustration of the OpenRM framework where the reward model, when receiving the candidate responses, progressively invokes external tools to gather useful evidence, and then make the final judgment.
• Figure 2: Illustration of the overlap between RewardBench and the training datasets of different reward models. The embeddings are visualized in 2D space. The orange points denote training data and blue points denote RewardBench instances. We observe substantial overlap in JudgeLRM, RRM, and RM-R1, whereas our dataset shows almost no intersection, ensuring a fairer and less biased evaluation .
• Figure 3: Training process of the models trained with different variants of the vanilla training supervision in Eq. \ref{['eq:reward']}. We plot average response length (red) and average reward (blue) over training steps, with final test set accuracy annotated. The results highlight distinct failure modes: (b) lazy searching under $R_{\text{EM}}$, (d) over-searching and reward hacking under $R_{\text{EM}} + R_{\text{tool}}$, while (a) the proposed composite reward achieves stable improvement and the best accuracy.