AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems

TL;DR

A novel multi-agent system, AutoUE, is proposed, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation.

Abstract

Automatically generating 3D games in commercial game engines remains a non-trivial challenge, as it involves complex engine-related workflows for generating assets such as scenes, blueprints, and code. To address this challenge, we propose a novel multi-agent system, AutoUE, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. In order to mitigate tool-use hallucinations in LLMs, we introduce a retrieval-augmented generation mechanism that grounds agents with relevant UE tool documentation. Additionally, we incorporate game design patterns and engine constraints into the code generation process to ensure the generation of correct and robust code. Furthermore, we design an automated play-testing pipeline that generates and executes runtime test commands, enabling systematic evaluation of dynamic behaviors. Finally, we construct a game generation dataset and conduct a series of experiments that demonstrate AutoUE's ability to generate 3D games end-to-end, and validate the effectiveness of these designs.

Figures (18)

• Figure 1: An example of game scene and gameplay.
• Figure 2: The overall framework of our proposed multi-agent system AutoUE. Blue arrows indicate the workflow.
• Figure 3: Visual comparison with SceneX. The first column shows scenes from SceneX; the second column shows the corresponding scenes in AutoUE; the third and fourth columns present AutoUE’s interactions within each scene.
• Figure 4: The descriptions and screenshots of all Easy games.
• Figure 5: The descriptions and screenshots of all Medium games.