EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis

TL;DR

EnvScaler introduces an automated, scalable framework to synthesize diverse, executable tool-interactive environments for training LLM agents. It combines SkelBuilder (topic mining, logic modeling, and dual-agent quality assessment) with ScenGenerator (state initialization, task generation, and rule-based trajectory verification) to produce thousands of environment-task combinations. Empirical results across three benchmarks show significant gains for Qwen3 models when trained with SFT and further improved by RL, demonstrating improved multi-turn, multi-tool reasoning and generalization. The work advances practical tool-learning for LLM agents while acknowledging biases and domain coverage limitations inherent to automated synthesis.

Abstract

Large language models (LLMs) are expected to be trained to act as agents in various real-world environments, but this process relies on rich and varied tool-interaction sandboxes. However, access to real systems is often restricted; LLM-simulated environments are prone to hallucinations and inconsistencies; and manually built sandboxes are hard to scale. In this paper, we propose EnvScaler, an automated framework for scalable tool-interaction environments via programmatic synthesis. EnvScaler comprises two components. First, SkelBuilder constructs diverse environment skeletons through topic mining, logic modeling, and quality evaluation. Then, ScenGenerator generates multiple task scenarios and rule-based trajectory validation functions for each environment. With EnvScaler, we synthesize 191 environments and about 7K scenarios, and apply them to Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Qwen3 series models. Results on three benchmarks show that EnvScaler significantly improves LLMs' ability to solve tasks in complex environments involving multi-turn, multi-tool interactions. We release our code and data at https://github.com/RUC-NLPIR/EnvScaler.

Figures (26)

• Figure 1: An illustration of tool-interactive environments. The environment (1) defines rules and provides tool interfaces to the agent; (2) executes the agent’s tool calls to update its state and return results.
• Figure 2: The overview of EnvScaler.
• Figure 3: The overall framework of SkelBuilder.
• Figure 4: The overall framework of ScenGenerator.
• Figure 5: Pairwise comparison of different LLMs on a random sample of 50 scenarios under Non-Conv setting.