Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

Jiacheng Guo, Ling Yang, Peter Chen, Qixin Xiao, Yinjie Wang, Xinzhe Juan, Jiahao Qiu, Ke Shen, Mengdi Wang

TL;DR

GenEnv tackles the data-collection bottleneck in training capable LLM agents by introducing a two-player, difficulty-aligned co-evolution between an Agent Policy and an Environment Policy. Task difficulty is adaptively tuned via an α-Curriculum Reward, creating a data-evolving training set that adaptively targets the agent's zone of proximal development. Theoretical results show that intermediate difficulty maximizes learning signals and that the curriculum ranking is statistically consistent, while experiments across five benchmarks demonstrate strong 7B performance and data efficiency rivaling larger models and static augmentation. This approach yields an emergent curriculum, data efficiency gains, and practical impact for scalable, interactive agent training in high-cost domains.

Abstract

Training capable Large Language Model (LLM) agents is critically bottlenecked by the high cost and static nature of real-world interaction data. We address this by introducing GenEnv, a framework that establishes a difficulty-aligned co-evolutionary game between an agent and a scalable, generative environment simulator. Unlike traditional methods that evolve models on static datasets, GenEnv instantiates a dataevolving: the simulator acts as a dynamic curriculum policy, continuously generating tasks specifically tailored to the agent's ``zone of proximal development''. This process is guided by a simple but effective $α$-Curriculum Reward, which aligns task difficulty with the agent's current capabilities. We evaluate GenEnv on five benchmarks, including API-Bank, ALFWorld, BFCL, Bamboogle, and TravelPlanner. Across these tasks, GenEnv improves agent performance by up to \textbf{+40.3\%} over 7B baselines and matches or exceeds the average performance of larger models. Compared to Gemini 2.5 Pro-based offline data augmentation, GenEnv achieves better performance while using 3.3$\times$ less data. By shifting from static supervision to adaptive simulation, GenEnv provides a data-efficient pathway for scaling agent capabilities.

GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

TL;DR

GenEnv tackles the data-collection bottleneck in training capable LLM agents by introducing a two-player, difficulty-aligned co-evolution between an Agent Policy and an Environment Policy. Task difficulty is adaptively tuned via an α-Curriculum Reward, creating a data-evolving training set that adaptively targets the agent's zone of proximal development. Theoretical results show that intermediate difficulty maximizes learning signals and that the curriculum ranking is statistically consistent, while experiments across five benchmarks demonstrate strong 7B performance and data efficiency rivaling larger models and static augmentation. This approach yields an emergent curriculum, data efficiency gains, and practical impact for scalable, interactive agent training in high-cost domains.

Abstract

Training capable Large Language Model (LLM) agents is critically bottlenecked by the high cost and static nature of real-world interaction data. We address this by introducing GenEnv, a framework that establishes a difficulty-aligned co-evolutionary game between an agent and a scalable, generative environment simulator. Unlike traditional methods that evolve models on static datasets, GenEnv instantiates a dataevolving: the simulator acts as a dynamic curriculum policy, continuously generating tasks specifically tailored to the agent's ``zone of proximal development''. This process is guided by a simple but effective -Curriculum Reward, which aligns task difficulty with the agent's current capabilities. We evaluate GenEnv on five benchmarks, including API-Bank, ALFWorld, BFCL, Bamboogle, and TravelPlanner. Across these tasks, GenEnv improves agent performance by up to \textbf{+40.3\%} over 7B baselines and matches or exceeds the average performance of larger models. Compared to Gemini 2.5 Pro-based offline data augmentation, GenEnv achieves better performance while using 3.3 less data. By shifting from static supervision to adaptive simulation, GenEnv provides a data-efficient pathway for scaling agent capabilities.
Paper Structure (42 sections, 2 theorems, 28 equations, 8 figures, 2 tables, 1 algorithm)

This paper contains 42 sections, 2 theorems, 28 equations, 8 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. GenEnv: Difficulty-Aligned Co-Evolution
  3. Data-Evolving Paradigm: From Static Corpora to Adaptive Simulation
  4. Rewards: Agent vs. Environment (with explicit equation references)
  5. Agent Task Reward ($R_{\text{agent}}$)
  6. Environment Difficulty-Alignment Reward ($R_{\text{env}}$)
  7. Data Structures and How New Training Data Is Produced
  8. What the environment generates.
  9. What the agent produces.
  10. Two growing datasets: one for the agent, one for the environment.
  11. Two growing datasets: one for the agent, one for the environment.
  12. How this produces a "data-evolving" training set.
  13. Two-Player Curriculum RL: Optimization Loop
  14. Player 1 Update (Agent).
  15. Player 2 Update (Environment).
  16. ...and 27 more sections

Key Result

Proposition 1

Suppose Assumption assump:bounded-score holds and the baseline is chosen as $b(\tau) = p(\tau)$. Then there exist positive constants $C_{\min}$ and $C_{\max}$, independent of $p(\tau)$, such that In particular, up to constant factors, the expected squared gradient norm is proportional to $p(\tau)(1 - p(\tau))$, which is maximized when $p(\tau) = 1/2$, i.e., for tasks of intermediate difficulty.

Figures (8)

  • Figure 1: GenEnv’s cross-benchmark gains and data efficiency. (a) We compare GenEnv (7B) against representative baselines (Qwen2.5-7B, ReSearch, SearchR1, ToRL) and larger open models (e.g., Qwen3-14B, GPT-OSS-20B). Blue callouts report the absolute improvement of GenEnv over Qwen2.5-7B on each benchmark. (b) Validation data-efficiency comparison on BFCL: GenEnv surpasses RandomEnv and Static Augmentation, and outperforms Gemini-based offline augmentation even with 3.3$\times$ more synthetic data. Together, the figure shows that difficulty-aligned adaptive simulation can outperform stronger static augmentation baselines under comparable training settings.
  • Figure 2: A comparison between the traditional training paradigm and our proposed GenEnv framework. The traditional approach (top) relies on high-cost interaction with the real world to create a static dataset, leading to inefficient training and poor generalization. GenEnv (bottom) creates a co-evolutionary loop where an Environment LLM generates adaptive tasks for the Agent LLM, enabling low-cost simulation, an adaptive curriculum, and improved efficiency.
  • Figure 3: The GenEnv Co-Evolutionary Loop. (1) The Environment Policy generates tasks. (2) The Agent Policy attempts them. (3) The environment reward (difficulty alignment) updates the simulator, while the agent reward (task success) updates the agent.
  • Figure 4: Training dynamics of GenEnv. From left to right: (a) training step-wise reward (critic/score/mean); (b) validation score across epochs; (c) batch-level ground-truth accuracy; and (d) per-epoch average reward. The curves show that GenEnv trains stably without reward collapse or divergence, with both reward and accuracy improving smoothly over time.
  • Figure 5: Emergent curriculum in GenEnv. Across training epochs, the environment simulator gradually increases task complexity (a), reflected by longer task descriptions; the agent correspondingly produces longer reasoning chains (b) as it learns to solve harder tasks; and its success rate (c) remains within a controlled band despite rising difficulty. Together these curves show that GenEnv induces an emergent curriculum in which task difficulty and agent capability co-evolve in a stable manner.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Proposition 1: Intermediate difficulty maximizes gradient signal
  • proof : Proof sketch
  • Remark 1: $\frac{1}{2}$-Curriculum reward
  • Theorem 1: Ranking consistency of $R_{\text{env}}$
  • proof : Proof sketch