DreamGarden: A Designer Assistant for Growing Games from a Single Prompt

TL;DR

DreamGarden introduces a multi-agent, LLM-driven planner that decomposes a single high-level prompt into a hierarchical plan and delegates leaf tasks to specialized submodules to generate playable Unreal Engine prototypes. The system emphasizes a node-based GUI and human-in-the-loop interactions to balance autonomy and designer oversight during rapid prototyping. Through an in-person usability study, the authors reveal that intermediate planning artifacts are valuable for understanding and guiding development, though complex tasks still challenge full autonomy. The work demonstrates a novel interaction paradigm for open-ended game design and outlines design and ethical considerations for future semi-autonomous design tools with broader applicability beyond gaming.

Abstract

Coding assistants are increasingly leveraged in game design, both generating code and making high-level plans. To what degree can these tools align with developer workflows, and what new modes of human-computer interaction can emerge from their use? We present DreamGarden, an AI system capable of assisting with the development of diverse game environments in Unreal Engine. At the core of our method is an LLM-driven planner, capable of breaking down a single, high-level prompt -- a dream, memory, or imagined scenario provided by a human user -- into a hierarchical action plan, which is then distributed across specialized submodules facilitating concrete implementation. This system is presented to the user as a garden of plans and actions, both growing independently and responding to user intervention via seed prompts, pruning, and feedback. Through a user study, we explore design implications of this system, charting courses for future work in semi-autonomous assistants and open-ended simulation design.

Figures (10)

• Figure 1: Growth of a scene in DreamGarden. The initial task prompts the procedural mesh generation submodule to write code using Perlin noise to produce hilly terrain (left). A followup task requests foliage to be added to this terrain (middle) which is rendered more diverse after visual feedback (right).
• Figure 2: Example of autonomous scene-generation on DreamGarden. Here, the system iterates on a scene involving chickens laying golden eggs by writing and extending C++ Unreal Actor classes, beginning with some initial terrain and foliage, then adding dynamically-spawning and physics-enabled golden eggs, and finally chickens, using meshes that it has generated/downloaded at prior iterations.
• Figure 3: The planning module converts the user's open-ended seed prompt into a broad plan for designing a simulation in Unreal Engine, then recursively breaks this plan down into more fine-grained steps, eventually terminating in leaf nodes which are then converted to concrete implementation tasks for the available implementation submodules.
• Figure 4: The code generation submodule pipeline. C++ code for Actors in Unreal Engine is generated given an implementation task prompt. Feedback is generated given any errors resulting from compilation, python editor scripting, or runtime.
• Figure 5: Submodules for generating/downloading meshes.