Procedural Game Level Design with Deep Reinforcement Learning

TL;DR

This work addresses procedural level design by embedding a co-adaptive DRL framework in Unity, where a hummingbird agent learns to collect nectar while a separate island agent learns to generate context-aware flower layouts. The approach employs Proximal Policy Optimization with a rich set of auxiliary observations and a dynamic feedback loop that couples environment generation to agent performance. Key contributions include the incorporation of auxiliary inputs, a penalty-based layout evaluation, and extensive environment randomization that yields robust generalization and emergent strategies such as altitude-aware navigation. The results demonstrate that DRL can drive both content generation and gameplay behavior, offering a scalable pathway toward autonomous, adaptive game level design.

Abstract

Procedural content generation (PCG) has become an increasingly popular technique in game development, allowing developers to generate dynamic, replayable, and scalable environments with reduced manual effort. In this study, a novel method for procedural level design using Deep Reinforcement Learning (DRL) within a Unity-based 3D environment is proposed. The system comprises two agents: a hummingbird agent, acting as a solver, and a floating island agent, responsible for generating and placing collectible objects (flowers) on the terrain in a realistic and context-aware manner. The hummingbird is trained using the Proximal Policy Optimization (PPO) algorithm from the Unity ML-Agents toolkit. It learns to navigate through the terrain efficiently, locate flowers, and collect them while adapting to the ever-changing procedural layout of the island. The island agent is also trained using the Proximal Policy Optimization (PPO) algorithm. It learns to generate flower layouts based on observed obstacle positions, the hummingbird's initial state, and performance feedback from previous episodes. The interaction between these agents leads to emergent behavior and robust generalization across various environmental configurations. The results demonstrate that the approach not only produces effective and efficient agent behavior but also opens up new opportunities for autonomous game level design driven by machine learning. This work highlights the potential of DRL in enabling intelligent agents to both generate and solve content in virtual environments, pushing the boundaries of what AI can contribute to creative game development processes.

Figures (9)

• Figure 1: System architecture showing the interaction between the procedural island generator and the hummingbird reinforcement learning agent.
• Figure 2: Unity environment setup featuring the floating island terrain. The scene includes procedurally placed flowers, static obstacles like rocks and bushes, and the hummingbird agent.
• Figure 3: Unity scene used for training. The floating island terrain contains trees, rocks, and procedurally placed flower prefabs.
• Figure 4: Reward per timestep over training. Shaded area indicates standard deviation across 8 parallel environments.
• Figure 5: Visualization of the hummingbird agent’s ray perception system. The agent uses rays to detect objects in its environment, including flowers, obstacles, and terrain. Additional observation inputs—such as velocity, rotation, terrain normals, and procedural spawn parameters—are encoded numerically and not visualized here.