Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Reasoning Reward Model for Agents

Kaixuan Fan, Kaituo Feng, Manyuan Zhang, Tianshuo Peng, Zhixun Li, Yilei Jiang, Shuang Chen, Peng Pei, Xunliang Cai, Xiangyu Yue

TL;DR

Sparse final-reward signals hinder long-horizon agentic reasoning. The authors propose Agent-RRM, a multi-faceted reward model that provides explicit reasoning traces, targeted critiques, and holistic scores, enabling dense, reasoning-aware supervision. Integrating Agent-RRM into agents via three strategies (Reagent-C, Reagent-R, Reagent-U) yields substantial gains across 12 benchmarks, with Reagent-U achieving notable improvements on GAIA and WebWalkerQA and strong cross-modal performance. The work demonstrates that combining textual critique with model-based rewards effectively guides complex tool use and multimodal reasoning, offering a scalable roadmap for future agentic RL research.

Abstract

Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.

Exploring Reasoning Reward Model for Agents

TL;DR

Sparse final-reward signals hinder long-horizon agentic reasoning. The authors propose Agent-RRM, a multi-faceted reward model that provides explicit reasoning traces, targeted critiques, and holistic scores, enabling dense, reasoning-aware supervision. Integrating Agent-RRM into agents via three strategies (Reagent-C, Reagent-R, Reagent-U) yields substantial gains across 12 benchmarks, with Reagent-U achieving notable improvements on GAIA and WebWalkerQA and strong cross-modal performance. The work demonstrates that combining textual critique with model-based rewards effectively guides complex tool use and multimodal reasoning, offering a scalable roadmap for future agentic RL research.

Abstract

Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.
Paper Structure (47 sections, 9 equations, 6 figures, 4 tables)

This paper contains 47 sections, 9 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic Reinforcement Learning
  4. Reward Modeling
  5. Method
  6. Preliminaries: GRPO Framework
  7. Agentic Tool Design
  8. Dataset Construction
  9. Agent Training Datasets
  10. Reward Model Datasets
  11. Agent-RRM: Reward Model Training
  12. Reagent: Integrating Reasoning Rewards into Agents
  13. Textual-augmented Refinement
  14. Reward-augmented Guidance
  15. Unified Feedback Integration
  16. ...and 32 more sections

Figures (6)

  • Figure 1: Detailed distribution information of Reagent-SFT-55.6K and Reagent-RL-709K.
  • Figure 2: Overview of the Reagent training scheme. We explore three integration variants: Reagent-C (blue arrows), Reagent-R (gray arrows), and Reagent-U (both arrows).
  • Figure 3: Impact of Agent-RRM reward weight $\lambda$ on task performance.
  • Figure 4: The prompt used for generating structured judgments of reward model.
  • Figure 5: Case 1: a search question from GAIA.
  • ...and 1 more figures