Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ToolGym: an Open-world Tool-using Environment for Scalable Agent Testing and Data Curation

Ziqiao Xi, Shuang Liang, Qi Liu, Jiaqing Zhang, Letian Peng, Fang Nan, Meshal Nayim, Tianhui Zhang, Rishika Mundada, Lianhui Qin, Biwei Huang, Kun Zhou

TL;DR

ToolGym presents an open-world tool-using environment that scales to thousands of tools across hundreds of apps to realistically test and train LLM agents. It combines MCP-unified tool formatting, a task creation engine for long-horizon, constraint-rich workflows, and a state controller to inject failures, all under a planner–actor architecture that separates reasoning from execution. The work reveals a consistent misalignment between planning and execution in current models, highlights constraint-following as a key bottleneck, and identifies DeepSeek-v3.2 as particularly robust under disruptions, while showing data-efficient gains from 1,170 trajectories for fine-tuning. Overall, ToolGym serves as both a rigorous benchmark and a data engine for advancing tool-using agents, with code and data slated for public release.

Abstract

Tool-using LLM agents still struggle in open-world settings with large tool pools, long-horizon objectives, wild constraints, and unreliable tool states. For scalable and realistic training and testing, we introduce an open-world tool-using environment, built on 5,571 format unified tools across 204 commonly used apps. It includes a task creation engine that synthesizes long-horizon, multi-tool workflows with wild constraints, and a state controller that injects interruptions and failures to stress-test robustness. On top of this environment, we develop a tool select-then-execute agent framework with a planner-actor decomposition to separate deliberate reasoning and self-correction from step-wise execution. Comprehensive evaluation of state-of-the-art LLMs reveals the misalignment between tool planning and execution abilities, the constraint following weakness of existing LLMs, and DeepSeek-v3.2's strongest robustness. Finally, we collect 1,170 trajectories from our environment to fine-tune LLMs, achieving superior performance to baselines using 119k samples, indicating the environment's value as both a realistic benchmark and a data engine for tool-using agents. Our code and data will be publicly released.

ToolGym: an Open-world Tool-using Environment for Scalable Agent Testing and Data Curation

TL;DR

ToolGym presents an open-world tool-using environment that scales to thousands of tools across hundreds of apps to realistically test and train LLM agents. It combines MCP-unified tool formatting, a task creation engine for long-horizon, constraint-rich workflows, and a state controller to inject failures, all under a planner–actor architecture that separates reasoning from execution. The work reveals a consistent misalignment between planning and execution in current models, highlights constraint-following as a key bottleneck, and identifies DeepSeek-v3.2 as particularly robust under disruptions, while showing data-efficient gains from 1,170 trajectories for fine-tuning. Overall, ToolGym serves as both a rigorous benchmark and a data engine for advancing tool-using agents, with code and data slated for public release.

Abstract

Tool-using LLM agents still struggle in open-world settings with large tool pools, long-horizon objectives, wild constraints, and unreliable tool states. For scalable and realistic training and testing, we introduce an open-world tool-using environment, built on 5,571 format unified tools across 204 commonly used apps. It includes a task creation engine that synthesizes long-horizon, multi-tool workflows with wild constraints, and a state controller that injects interruptions and failures to stress-test robustness. On top of this environment, we develop a tool select-then-execute agent framework with a planner-actor decomposition to separate deliberate reasoning and self-correction from step-wise execution. Comprehensive evaluation of state-of-the-art LLMs reveals the misalignment between tool planning and execution abilities, the constraint following weakness of existing LLMs, and DeepSeek-v3.2's strongest robustness. Finally, we collect 1,170 trajectories from our environment to fine-tune LLMs, achieving superior performance to baselines using 119k samples, indicating the environment's value as both a realistic benchmark and a data engine for tool-using agents. Our code and data will be publicly released.
Paper Structure (44 sections, 2 figures, 6 tables)

This paper contains 44 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. ToolGym
  3. Extensible Toolset Construction
  4. MCP Server Selection.
  5. Tool Validation.
  6. Tool Retrieval Index.
  7. Task Creation Engine
  8. Tool Candidate Sampling.
  9. Synthesizing Task with Constraints.
  10. Task Evaluation Protocol.
  11. State Controller
  12. Agent Framework
  13. Agent Roles
  14. The Actor.
  15. The Planner.
  16. ...and 29 more sections

Figures (2)

  • Figure 1: The overall framework of ToolGym. The pipeline begins with curating real-world MCP tools and synthesizing tasks with wild constraints (Left). The agent employs a Planner--Actor architecture to decompose long-horizon goals, where the Actor interacts with the environment via a State Controller. Crucially, the Controller intercepts tool calls to inject realistic failures (e.g., timeouts, state changes) before execution on real servers. The resulting trajectories support both rigorous Evaluation and high-quality Data Curation for model training.
  • Figure 2: Five-dimensional personality radar charts of different MCP-based agents.