WorldGen: From Text to Traversable and Interactive 3D Worlds

TL;DR

WorldGen presents a modular pipeline that converts text prompts into interactive 3D worlds by integrating text-driven scene planning, navmesh-guided reconstruction, compositional scene decomposition, and per-object enhancement. It combines LLMs with procedural generation and diffusion-based 3D synthesis (AssetGen2/VecSet) to produce navigable, textured scenes that are immediately usable in standard game engines. Key innovations include navmesh-conditioned scene generation, accelerated AutoPartGen-based decomposition, and per-object image/mesh/texture refinement with multi-view texture synthesis and verification. The approach enables scalable, end-to-end world generation from language, with demonstrated improvements in geometric/navigational coherence and practical performance, suitable for gaming, simulation, and immersive social environments.

Abstract

We introduce WorldGen, a system that enables the automatic creation of large-scale, interactive 3D worlds directly from text prompts. Our approach transforms natural language descriptions into traversable, fully textured environments that can be immediately explored or edited within standard game engines. By combining LLM-driven scene layout reasoning, procedural generation, diffusion-based 3D generation, and object-aware scene decomposition, WorldGen bridges the gap between creative intent and functional virtual spaces, allowing creators to design coherent, navigable worlds without manual modeling or specialized 3D expertise. The system is fully modular and supports fine-grained control over layout, scale, and style, producing worlds that are geometrically consistent, visually rich, and efficient to render in real time. This work represents a step towards accessible, generative world-building at scale, advancing the frontier of 3D generative AI for applications in gaming, simulation, and immersive social environments.

Figures (37)

• Figure 1: World snapshots generated by WorldGen. Each scene consists of individually editable objects represented as fully textured 3D meshes. As explicit geometry, these worlds naturally support collision, and navigation—allowing characters to climb, jump, and interact. The resulting assets are immediately deployable in game engines.
• Figure 2: WorldGen overview. Our pipeline begins by planning the scene layout, producing a blockout ($B$), reference image ($\mathbf{R}$), and navigation mesh (S) (Stage 1). Next, we generate a single 3D mesh that aligns with this plan, preserving navigable areas and overall composition (Stage 2). The scene is then decomposed into individual entities (Stage 3), which are refined at higher resolution (Stage 4), resulting in a high-quality, traversable, and visually cohesive final scene.
• Figure 3: 3D Layout Generation. An LLM parses the input prompt into structured parameters (JSON) to drive a procedural generator, producing a coarse 3D blockout. This blockout is then rendered to depth, which conditions the generation of the final scene reference image.
• Figure 4: Depth-conditioned generation across density (columns) and verticality range (rows). In each grid cell, we show the input depth map (left) and the corresponding generated image conditioned on that depth (right). Columns are ordered by increasing density from left to right; rows are ordered by increasing verticality range from low to high.
• Figure 5: AssetGen2 and Navmesh architectures. Left: Overview of the base AssetGen mesh generation architecture. Right: Our Navmesh conditioned scene mesh generation (Stage II) based on cross-attention