Evolving Curricula with Regret-Based Environment Design

TL;DR

The paper tackles robust generalization in reinforcement learning by integrating evolution-inspired curriculum design with regret-based selection. It introduces ACCEL, a method that edits high-regret training levels to continually grow task difficulty aligned with the agent’s capabilities, while preserving the theoretical benefits of minimax regret frameworks. Across MiniGrid, MiniHack/POET-like mazes, and a multi-parameter BipedalWalker domain, ACCEL achieves strong zero-shot transfer and superior performance with substantially less compute than prior open-ended approaches like POET. The work demonstrates that regret-guided level editing can yield open-ended, generalist agents and outlines future directions for editors, diversity, and scalability to larger design spaces.

Abstract

It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at accelagent.github.io.

Figures (29)

• Figure 1: The evolution of a level in three different environments: MiniHack lava grids, MiniGrid mazes and BipedalWalker terrains. In each case, the far left shows a base level, acting as a parent for subsequent edited levels to the right. Each level along the evolutionary path has a high regret for the student agent at that point in time. Thus the level difficulty co-evolves with the agent's capabilities. In each environment, we see that despite starting with simple levels, the pursuit of high regret leads to increasingly complex challenges. This complexity emerges entirely without relying on any environment-specific exploration heuristics. Note that since the agent can move diagonally in the lava environment, the final level in the top row is solvable.
• Figure 2: An overview of ACCEL. Levels are randomly sampled from a generator and evaluated, with high-regret levels added to the level replay buffer. The curator selects levels to replay, and the student only trains on replay levels. After training, the regret of replayed levels are edited and evaluated again for level replay.
• Figure 3: ACCEL
• Figure 4: Emergent complexity metrics for mazes generated during training. Mean and standard error across 5 training seeds are shown.
• Figure 5: Aggregate zero-shot test performance in the maze domain.

Theorems & Definitions (1)

• Remark 1