Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Calling

Jianghao Lin, Yuanyuan Shi, Xin Peng, Renjie Ding, Hairui Wang, Yuxuan Peng, Bizhe Bai, Weixi Song, Fengshuo Bai, Huacan Chai, Weinan Zhang, Fei Huang, Ying Wen

TL;DR

This work tackles the challenge of scaling inference for structured outputs in LLM-based function calling by introducing ToolPRM, a fine-grained process reward model that supervises intra-call steps. It decomposes each tool invocation into semantic decisions (function-name selection, parameter naming, and value filling) and trains ToolPRM with step-level annotations derived from augmented datasets built on xLam function-calling resources. Coupled with fine-grained beam search, ToolPRM guides search toward valid, high-quality function calls, outperforming coarse-grained reward models and other inference-scaling baselines across benchmarks, particularly for smaller models and edge deployments. The paper also formalizes the principle "explore more but retain less" for structured outputs, showing that increasing exploration width while aggressively pruning trajectories yields more reliable tool use and efficient computation in structured generation tasks.

Abstract

Large language models (LLMs) are increasingly demonstrating strong capabilities as autonomous agents, with function calling serving as a core mechanism for interaction with the environment. Meanwhile, inference scaling has become a cutting-edge technique to enhance LLM performance by allocating more computational resources during the inference process. However, current research on inference scaling primarily focuses on unstructured output generation tasks, leaving its application in structured outputs, like function calling, largely underexplored. To bridge this gap, we propose an inference scaling framework that combines fine-grained beam search with a process reward model, ToolPRM, which scores the internal steps of each single function call. To train ToolPRM, we construct the first fine-grained intra-call process supervision dataset, automatically annotated with function-masking techniques to provide step-level rewards for structured tool-use reasoning. Extensive experiments demonstrate that ToolPRM beats the coarse-grained and outcome reward models in terms of predictive accuracy, indicating its stronger capability in supervising the function calling inference process. Inference scaling technique equipped with ToolPRM also significantly improves the backbone model performance across various function calling tasks and benchmarks. More importantly, we reveal a key principle for applying inference scaling techniques to structured outputs: "explore more but retain less" due to the unrecoverability characteristics of structured function calling generation.

ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Calling

TL;DR

This work tackles the challenge of scaling inference for structured outputs in LLM-based function calling by introducing ToolPRM, a fine-grained process reward model that supervises intra-call steps. It decomposes each tool invocation into semantic decisions (function-name selection, parameter naming, and value filling) and trains ToolPRM with step-level annotations derived from augmented datasets built on xLam function-calling resources. Coupled with fine-grained beam search, ToolPRM guides search toward valid, high-quality function calls, outperforming coarse-grained reward models and other inference-scaling baselines across benchmarks, particularly for smaller models and edge deployments. The paper also formalizes the principle "explore more but retain less" for structured outputs, showing that increasing exploration width while aggressively pruning trajectories yields more reliable tool use and efficient computation in structured generation tasks.

Abstract

Large language models (LLMs) are increasingly demonstrating strong capabilities as autonomous agents, with function calling serving as a core mechanism for interaction with the environment. Meanwhile, inference scaling has become a cutting-edge technique to enhance LLM performance by allocating more computational resources during the inference process. However, current research on inference scaling primarily focuses on unstructured output generation tasks, leaving its application in structured outputs, like function calling, largely underexplored. To bridge this gap, we propose an inference scaling framework that combines fine-grained beam search with a process reward model, ToolPRM, which scores the internal steps of each single function call. To train ToolPRM, we construct the first fine-grained intra-call process supervision dataset, automatically annotated with function-masking techniques to provide step-level rewards for structured tool-use reasoning. Extensive experiments demonstrate that ToolPRM beats the coarse-grained and outcome reward models in terms of predictive accuracy, indicating its stronger capability in supervising the function calling inference process. Inference scaling technique equipped with ToolPRM also significantly improves the backbone model performance across various function calling tasks and benchmarks. More importantly, we reveal a key principle for applying inference scaling techniques to structured outputs: "explore more but retain less" due to the unrecoverability characteristics of structured function calling generation.
Paper Structure (21 sections, 1 equation, 4 figures, 3 tables)

This paper contains 21 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Function Calling for LLM Agents
  4. Inference Scaling Strategies
  5. Process Reward Models
  6. Methodology
  7. Fine-Grained Decomposition of Function Calls
  8. Data Collection and Annotation
  9. Data Collection.
  10. Result Annotation.
  11. State Transition Mechanism and ToolPRM Training
  12. Beam Search with Fine-Grained Process Supervision
  13. Experiments
  14. Experiment Setups
  15. Datasets.
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The illustration of data collection for ToolPRM.
  • Figure 2: The state transition of function calling (left) and beam search with ToolPRM (right).
  • Figure 3: The learning curves of different reward models (i.e., ORM, C-PRM, ToolPRM) on five-epoch training, in terms of loss, step-level accuracy, and trajectory-level accuracy.
  • Figure 4: The performance on BFCL (Averaged Accuracy) and ToolAlpaca (Averaged F1 Score) w.r.t. different generation budgets and different inference scaling strategies. We conduct experiments on Hammer2.1-1.5B and Hammer2.1-3B policy models.