Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning

Shihao Cai, Runnan Fang, Jialong Wu, Baixuan Li, Xinyu Wang, Yong Jiang, Liangcai Su, Liwen Zhang, Wenbiao Yin, Zhen Zhang, Fuli Feng, Pengjun Xie, Xiaobin Wang

TL;DR

AutoForge tackles the cost, fragility, and limited breadth of agentic RL by coupling an automated, scalable synthesis of interactive environments with Environment-level Relative Policy Optimization (ERPO). The pipeline automatically constructs environments from tool descriptions, generates complex tool-sequences within a DAG framework, and creates verifiable tasks, while masking erroneous user behavior and estimating advantages at the environment level to boost stability. Empirical results on tau-bench, tau2-Bench, VitaBench, and ACEBench-zh show strong in-domain performance gains and notable out-of-domain generalization, with RL-based variants benefiting most from the synthetic environments. This work demonstrates that automated mock environments can substantially advance scalable, robust agentic learning and generalization to unseen tasks.

Abstract

Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.

AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning

TL;DR

AutoForge tackles the cost, fragility, and limited breadth of agentic RL by coupling an automated, scalable synthesis of interactive environments with Environment-level Relative Policy Optimization (ERPO). The pipeline automatically constructs environments from tool descriptions, generates complex tool-sequences within a DAG framework, and creates verifiable tasks, while masking erroneous user behavior and estimating advantages at the environment level to boost stability. Empirical results on tau-bench, tau2-Bench, VitaBench, and ACEBench-zh show strong in-domain performance gains and notable out-of-domain generalization, with RL-based variants benefiting most from the synthetic environments. This work demonstrates that automated mock environments can substantially advance scalable, robust agentic learning and generalization to unseen tasks.

Abstract

Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.
Paper Structure (27 sections, 4 equations, 4 figures, 3 tables)

This paper contains 27 sections, 4 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic Reinforcement Learning.
  4. Environment Emulation.
  5. Method
  6. Environment Synthesis
  7. Tool‑Sequence Generation
  8. Task Generation
  9. ERPO
  10. Experiments
  11. Experimental Setup
  12. Benchmarks.
  13. Baselines.
  14. Implementation Details.
  15. Performance Comparison (RQ1 & RQ2)
  16. ...and 12 more sections

Figures (4)

  • Figure 1: The AutoForge framework consists of a unified pipeline for scalable synthesis of simulated environments and high‑difficulty, easily verifiable tasks, and ERPO algorithm for multi‑environment agentic RL.
  • Figure 2: Out-of-Domain performance on the ACEBench-zh. We reported both the Agent and Overall subset scores.
  • Figure 3:
  • Figure 4: Effectiveness of interleaved thinking.