Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autoverse: An Evolvable Game Language for Learning Robust Embodied Agents

Sam Earle, Julian Togelius

TL;DR

Autoverse presents a scalable, evolvable DSL for open-ended learning in 2D grid environments, leveraging cellular automata–style rewrite rules implemented as convolutions to enable GPU-accelerated, differentiable simulation. The framework combines a warm-start phase of imitation learning from search trajectories with an adaptive open-ended RL loop (PPO) in evolving environments, yielding a curriculum of increasingly challenging tasks. Key contributions include a detailed batched engine with evolvable components, a mutation-centric search objective (maximizing best-first search steps to a high-reward solution), and a distillation step that seeds RL with expert trajectories. The approach demonstrates improved policy generality and reveals a spectrum of dynamic environment behaviors, from chaotic to semi-stable, with implications for designing robust agents and future human-aligned or foundation-model–assisted open-ended learning pipelines.

Abstract

We introduce Autoverse, an evolvable, domain-specific language for single-player 2D grid-based games, and demonstrate its use as a scalable training ground for Open-Ended Learning (OEL) algorithms. Autoverse uses cellular-automaton-like rewrite rules to describe game mechanics, allowing it to express various game environments (e.g. mazes, dungeons, sokoban puzzles) that are popular testbeds for Reinforcement Learning (RL) agents. Each rewrite rule can be expressed as a series of simple convolutions, allowing for environments to be parallelized on the GPU, thereby drastically accelerating RL training. Using Autoverse, we propose jump-starting open-ended learning by imitation learning from search. In such an approach, we first evolve Autoverse environments (their rules and initial map topology) to maximize the number of iterations required by greedy tree search to discover a new best solution, producing a curriculum of increasingly complex environments and playtraces. We then distill these expert playtraces into a neural-network-based policy using imitation learning. Finally, we use the learned policy as a starting point for open-ended RL, where new training environments are continually evolved to maximize the RL player agent's value function error (a proxy for its regret, or the learnability of generated environments), finding that this approach improves the performance and generality of resultant player agents.

Autoverse: An Evolvable Game Language for Learning Robust Embodied Agents

TL;DR

Autoverse presents a scalable, evolvable DSL for open-ended learning in 2D grid environments, leveraging cellular automata–style rewrite rules implemented as convolutions to enable GPU-accelerated, differentiable simulation. The framework combines a warm-start phase of imitation learning from search trajectories with an adaptive open-ended RL loop (PPO) in evolving environments, yielding a curriculum of increasingly challenging tasks. Key contributions include a detailed batched engine with evolvable components, a mutation-centric search objective (maximizing best-first search steps to a high-reward solution), and a distillation step that seeds RL with expert trajectories. The approach demonstrates improved policy generality and reveals a spectrum of dynamic environment behaviors, from chaotic to semi-stable, with implications for designing robust agents and future human-aligned or foundation-model–assisted open-ended learning pipelines.

Abstract

We introduce Autoverse, an evolvable, domain-specific language for single-player 2D grid-based games, and demonstrate its use as a scalable training ground for Open-Ended Learning (OEL) algorithms. Autoverse uses cellular-automaton-like rewrite rules to describe game mechanics, allowing it to express various game environments (e.g. mazes, dungeons, sokoban puzzles) that are popular testbeds for Reinforcement Learning (RL) agents. Each rewrite rule can be expressed as a series of simple convolutions, allowing for environments to be parallelized on the GPU, thereby drastically accelerating RL training. Using Autoverse, we propose jump-starting open-ended learning by imitation learning from search. In such an approach, we first evolve Autoverse environments (their rules and initial map topology) to maximize the number of iterations required by greedy tree search to discover a new best solution, producing a curriculum of increasingly complex environments and playtraces. We then distill these expert playtraces into a neural-network-based policy using imitation learning. Finally, we use the learned policy as a starting point for open-ended RL, where new training environments are continually evolved to maximize the RL player agent's value function error (a proxy for its regret, or the learnability of generated environments), finding that this approach improves the performance and generality of resultant player agents.
Paper Structure (11 sections, 3 equations, 3 figures, 2 tables)

This paper contains 11 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Autoverse: a batched game engine with evolvable components
  4. Possible extensions to the Autoverse language
  5. Warm-starting open-ended learning from search
  6. Evolving game environments to maximize search-based complexity
  7. Imitation learning: distilling search-based solutions
  8. Open-ended reinforcement learning in evolving environments
  9. Results
  10. Related work
  11. Conclusion

Figures (3)

  • Figure 1: Examples of environment dynamics in environments evolved for maximum search depth. The player (blue tile) takes the best sequence of actions as returned by a greedy tree search algorithm in order to maximize the reward returned by the environment's transition rules.
  • Figure 2: An overview of autoverse's approach to generating novel environments and trajectories.
  • Figure 3: The trajectories and environments generated by autoverse are incorporated in an Open-Ended Reinforcement Learning loop.