Enhancing the Hierarchical Environment Design via Generative Trajectory Modeling

TL;DR

This paper tackles the problem of designing efficient training curricula for reinforcement learning agents under limited resources by introducing SHED, a hierarchical MDP framework where an upper-level teacher curates environments for a lower-level student. A key novelty is the integration of diffusion probabilistic models to generate synthetic trajectories, which accelerates teacher training by reducing costly real-environment interactions. The method explicitly represents the student’s capabilities as a vector across multiple evaluation environments and uses a tractable reward that promotes both improvement and fairness across diverse settings. Empirical results on Lunar Lander and BipedalWalker show SHED outperforms domain randomization, ACCEL, PAIRED, and h-MDP in terms of generalization and stability, demonstrating the practical value of diffusion-augmented, resource-conscious UED.

Abstract

Unsupervised Environment Design (UED) is a paradigm for automatically generating a curriculum of training environments, enabling agents trained in these environments to develop general capabilities, i.e., achieving good zero-shot transfer performance. However, existing UED approaches focus primarily on the random generation of environments for open-ended agent training. This is impractical in scenarios with limited resources, such as the constraints on the number of generated environments. In this paper, we introduce a hierarchical MDP framework for environment design under resource constraints. It consists of an upper-level RL teacher agent that generates suitable training environments for a lower-level student agent. The RL teacher can leverage previously discovered environment structures and generate environments at the frontier of the student's capabilities by observing the student policy's representation. Moreover, to reduce the time-consuming collection of experiences for the upper-level teacher, we utilize recent advances in generative modeling to synthesize a trajectory dataset to train the teacher agent. Our proposed method significantly reduces the resource-intensive interactions between agents and environments and empirical experiments across various domains demonstrate the effectiveness of our approach.

Figures (8)

• Figure 1: The overall framework of SHED.
• Figure 2: The distribution of the real $[s^\prime_1,s^\prime_2,s^\prime_3]$(red) and the synthetic $[s^\prime_1,s^\prime_2,s^\prime_3]$(blue) giving the fixed $(s^u,a^u)$. Specifically, the noise $\varepsilon$ in $f(s^u, a^u)$ is (i). left figure: $\varepsilon=\epsilon$, (ii). middle figure: $\varepsilon=3*\epsilon$, (iii). right figure: $\varepsilon=10*\epsilon$, where $\epsilon~\sim \mathcal{N}(0,1)$.
• Figure 3: Left: The average zero-shot transfer performances on the test environments in the Lunar lander environment (mean and standard error). Right: The average zero-shot transfer performances on the test environments in the BipedalWalker (mean and standard error).
• Figure 4: The distribution of the real $s^\prime$ and the synthetic $s^\prime$ conditioned on $(s,a)$.
• Figure 5: Left: The ablation study in the Lunar lander environment which investigates the effect of the size of the evaluation environment set. We provide the average zero-shot transfer performances on the test environments (mean and standard error). Right: Zero-shot transfer performance on the test environments under a longer time horizon in Lunar lander environments(mean and standard error).

Theorems & Definitions (2)

• Theorem 3.1
• proof