Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo

TL;DR

This work introduces ToolPRMBench, a large-scale benchmark for evaluating process reward models in tool-using agents by converting trajectories from diverse tool-using benchmarks into step-level samples with correct/incorrect actions and tool metadata. It combines offline and online trajectory sampling with a multi-LLM verification pipeline to produce high-quality supervision, and it investigates three ToolPRM variants (Base, CoT, GRPO). Across 17 models, ToolPRMBench reveals that tool-specific PRMs, especially those trained with reinforcement learning (GRPO), can outperform general PRMs and many open-source baselines, while API-based LLMs still lead in overall performance but at higher costs. The study also provides meta-evaluation, data synthesis analyses, and cost assessments, highlighting the importance of robust generalization (ID vs. OOD) and practical considerations for deploying reward-guided search in tool-using systems. Overall, ToolPRMBench offers a standardized, informative framework to advance reliable, scalable PRMs for real-world tool-enabled agents.

Abstract

Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

TL;DR

This work introduces ToolPRMBench, a large-scale benchmark for evaluating process reward models in tool-using agents by converting trajectories from diverse tool-using benchmarks into step-level samples with correct/incorrect actions and tool metadata. It combines offline and online trajectory sampling with a multi-LLM verification pipeline to produce high-quality supervision, and it investigates three ToolPRM variants (Base, CoT, GRPO). Across 17 models, ToolPRMBench reveals that tool-specific PRMs, especially those trained with reinforcement learning (GRPO), can outperform general PRMs and many open-source baselines, while API-based LLMs still lead in overall performance but at higher costs. The study also provides meta-evaluation, data synthesis analyses, and cost assessments, highlighting the importance of robust generalization (ID vs. OOD) and practical considerations for deploying reward-guided search in tool-using systems. Overall, ToolPRMBench offers a standardized, informative framework to advance reliable, scalable PRMs for real-world tool-enabled agents.

Abstract

Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.
Paper Structure (32 sections, 5 equations, 10 figures, 4 tables)

This paper contains 32 sections, 5 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool-using Benchmarks
  4. Reward-guided Search and Reward Modeling
  5. ToolPRMBench
  6. Trajectory Sampling
  7. Data Verification
  8. ToolPRM Training
  9. ToolPRM-Base.
  10. ToolPRM-CoT.
  11. ToolPRM-GRPO.
  12. Dataset Statistics
  13. Experiment
  14. Experiment Setting
  15. Models.
  16. ...and 17 more sections

Figures (10)

  • Figure 1: The overview pipeline of ToolPRMBench, including trajectory sampling, data verification & filtering, and ToolPRM training.
  • Figure 2: Statistics of trajectory length and number of functions in ToolPRMBench.
  • Figure 3: Statistics of error and category distribution in ToolPRMBench.
  • Figure 4: ToolPRM leaderboard on 17 LLMs.
  • Figure 5: Scaling analysis result on Qwen3 and LLaMA-3.
  • ...and 5 more figures