Improving Regret Approximation for Unsupervised Dynamic Environment Generation

TL;DR

This work tackles generalization gaps in reinforcement learning by reframing Unsupervised Environment Design (UED) as a regret-based curriculum problem and introducing Dynamic Environment Generation (DEGen). DEGen generates level components on-the-fly during agent exploration, delivering denser training signals and mitigating credit-assignment difficulties, while Maximised Negative Advantage (MNA) provides a more reliable regret metric than PVL or MaxMC. Together, DEGen and MNA yield substantial performance gains over replay-based and prior learned-generator methods, especially as environment size and complexity grow, and maintain competitive results in simpler settings. The approach advances scalable, open-ended curriculum generation for RL and points toward broader applicability in larger, more complex domains, including potential world-model extensions.

Abstract

Unsupervised Environment Design (UED) seeks to automatically generate training curricula for reinforcement learning (RL) agents, with the goal of improving generalisation and zero-shot performance. However, designing effective curricula remains a difficult problem, particularly in settings where small subsets of environment parameterisations result in significant increases in the complexity of the required policy. Current methods struggle with a difficult credit assignment problem and rely on regret approximations that fail to identify challenging levels, both of which are compounded as the size of the environment grows. We propose Dynamic Environment Generation for UED (DEGen) to enable a denser level generator reward signal, reducing the difficulty of credit assignment and allowing for UED to scale to larger environment sizes. We also introduce a new regret approximation, Maximised Negative Advantage (MNA), as a significantly improved metric to optimise for, that better identifies more challenging levels. We show empirically that MNA outperforms current regret approximations and when combined with DEGen, consistently outperforms existing methods, especially as the size of the environment grows. We have made all our code available here: https://github.com/HarryMJMead/Dynamic-Environment-Generation-for-UED.

Figures (22)

• Figure 1: Examples of two possible randomly generated levels. In the first, the agent (red triangle) can simply navigate to the goal (green square), whereas in the second, it is required to first obtain the key (two blue triangles) in order to unlock the door (blue unfilled square) blocking the path to the goal
• Figure 2: Illustration of how DEGen generates the level as the student agent explores the level, with purple colouring indicating areas that are yet to be observed.
• Figure 3: Minigrid zero-shot performance on hand-designed test set, showing mean and standard error across 8 runs.
• Figure 4: Minigrid with key and locked door zero-shot performance on hand-designed test set, trained on 13x13 training levels, showing mean and standard error across 8 runs.
• Figure 5: Minigrid with key and locked door zero-shot performance on hand-designed test set, trained on larger training levels, showing mean and standard error across 8 runs.