Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing

Yuwen Li, Wei Zhang, Zelong Huang, Mason Yang, Jiajun Wu, Shawn Guo, Huahao Hu, Lingyi Sun, Jian Yang, Mingjie Tang, Byran Dai

TL;DR

InfTool introduces a fully autonomous, self-evolving framework that uses a three-agent system to synthesize tool-use data directly from MCP API specifications, addressing data scarcity, generalization, and data-quality ceilings. The core engine, GRPO with gated rewards, iteratively refines the model by training on synthetic trajectories, which in turn enables the agents to produce higher-quality data. Empirical results on BFCL show InfTool-32B achieving 70.9% total accuracy, surpassing many open-source baselines and rivaling proprietary models, all without human annotation. This approach demonstrates that dynamic, self-generated training environments can unlock substantial tool-use capabilities in comparatively small models, with significant implications for scalable autonomous agents.

Abstract

Enabling Large Language Models (LLMs) to reliably invoke external tools remains a critical bottleneck for autonomous agents. Existing approaches suffer from three fundamental challenges: expensive human annotation for high-quality trajectories, poor generalization to unseen tools, and quality ceilings inherent in single-model synthesis that perpetuate biases and coverage gaps. We introduce InfTool, a fully autonomous framework that breaks these barriers through self-evolving multi-agent synthesis. Given only raw API specifications, InfTool orchestrates three collaborative agents (User Simulator, Tool-Calling Assistant, and MCP Server) to generate diverse, verified trajectories spanning single-turn calls to complex multi-step workflows. The framework establishes a closed loop: synthesized data trains the model via Group Relative Policy Optimization (GRPO) with gated rewards, the improved model generates higher-quality data targeting capability gaps, and this cycle iterates without human intervention. Experiments on the Berkeley Function-Calling Leaderboard (BFCL) demonstrate that InfTool transforms a base 32B model from 19.8% to 70.9% accuracy (+258%), surpassing models 10x larger and rivaling Claude-Opus, and entirely from synthetic data without human annotation.

Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing

TL;DR

InfTool introduces a fully autonomous, self-evolving framework that uses a three-agent system to synthesize tool-use data directly from MCP API specifications, addressing data scarcity, generalization, and data-quality ceilings. The core engine, GRPO with gated rewards, iteratively refines the model by training on synthetic trajectories, which in turn enables the agents to produce higher-quality data. Empirical results on BFCL show InfTool-32B achieving 70.9% total accuracy, surpassing many open-source baselines and rivaling proprietary models, all without human annotation. This approach demonstrates that dynamic, self-generated training environments can unlock substantial tool-use capabilities in comparatively small models, with significant implications for scalable autonomous agents.

Abstract

Enabling Large Language Models (LLMs) to reliably invoke external tools remains a critical bottleneck for autonomous agents. Existing approaches suffer from three fundamental challenges: expensive human annotation for high-quality trajectories, poor generalization to unseen tools, and quality ceilings inherent in single-model synthesis that perpetuate biases and coverage gaps. We introduce InfTool, a fully autonomous framework that breaks these barriers through self-evolving multi-agent synthesis. Given only raw API specifications, InfTool orchestrates three collaborative agents (User Simulator, Tool-Calling Assistant, and MCP Server) to generate diverse, verified trajectories spanning single-turn calls to complex multi-step workflows. The framework establishes a closed loop: synthesized data trains the model via Group Relative Policy Optimization (GRPO) with gated rewards, the improved model generates higher-quality data targeting capability gaps, and this cycle iterates without human intervention. Experiments on the Berkeley Function-Calling Leaderboard (BFCL) demonstrate that InfTool transforms a base 32B model from 19.8% to 70.9% accuracy (+258%), surpassing models 10x larger and rivaling Claude-Opus, and entirely from synthetic data without human annotation.
Paper Structure (41 sections, 7 equations, 20 figures, 5 tables)

This paper contains 41 sections, 7 equations, 20 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. Tool Use and MCP
  5. Role-Playing Multi-Agent Data Synthesis
  6. Framework Overview
  7. MCP Tree
  8. Hierarchical Data Synthesis
  9. Quality Assurance Mechanism
  10. Self-Evolution Training Loop
  11. Complexity Stratification
  12. Gated Composite Reward Policy Optimization
  13. Adaptive Data Injection
  14. Data Analysis
  15. Computational Complexity Analysis
  16. ...and 26 more sections

Figures (20)

  • Figure 1: MCP-based tool calling across three scenarios.
  • Figure 2: Overview of our framework. Step 1: Build MCP tool definitions from raw APIs. Step 2: Cold start Supervised Fine Tuning (SFT) sft. Step 3: Rollout evaluation to categorize samples. Step 4: GRPO training. Step 5-6: Multi-agent role-play (User, Agent, Server) generates complex trajectories. Step 7: Self-reflection filtering.
  • Figure 3: Token consumption distribution across tool-call categories and conversation lengths. The left panel shows token requirements for different tool-call frequencies (no calls, single call, multiple calls), while the right panel contrasts single-turn versus multi-turn conversations. Box plots indicate quartiles, with whiskers extending to the 5th and 95th percentiles.
  • Figure 4: Domain distribution in the training dataset.
  • Figure 5: Distribution of Conversation Turns.
  • ...and 15 more figures