Adversarial Environment Design via Regret-Guided Diffusion Models

TL;DR

A novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD), which can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy.

Abstract

Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments. Project page: https://rllab-snu.github.io/projects/ADD

Figures (14)

• Figure 1: Overview of ADD. After the agent is trained on environments produced by the environment generator, the environment critic is updated using the episodic results. Then, the environment critic guides the diffusion-based environment generator with the regret to produce adversarial environments. By repeating this process, the agent learns a policy that is robust to environmental changes.
• Figure 2: Partially observable navigation results.(a): Zero-shot performance on the 12 test environments. We report results across five random seeds, each evaluated over 100 independent episodes per environment. (b): Training curves on two challenging test environments. (c): Complexity metrics of the generated environments during training. (d): t-SNE embedding of the generated environments during training.
• Figure 3: 2D bipedal locomotion task results.(a): Zero-shot performance on the six test environments. We report results across five random seeds, each evaluated over 100 independent episodes per environment. (b): Complexity metrics of the generated environments and episodic return achieved during training.
• Figure 4: Controllable generation results for the partially observable navigation task. The figure shows the results of guiding the generator to generate progressively more difficult environments. We note that each row is generated from the same initial noise $\theta_T$.
• Figure 5: Maze environment generation using diffusion models. We represent the maze environment with a parameter $\theta \in \mathbb{R}^{13 \times 13 \times 3}$, with each channel indicating the location of walls, the agent, and the goal. After training the diffusion-based environment generator on a dataset of randomly generated environment parameters, we can sample maze environments by solving the reverse process (\ref{['backward approx sde']}).

Theorems & Definitions (8)

• Proposition 4.1
• Lemma A.1
• proof
• Lemma A.2
• proof
• Lemma A.3
• proof
• Proposition A.4