3D Scene Generation: A Survey

TL;DR

This survey consolidates the rapid progress in 3D scene generation by organizing methods into four paradigms—procedural, neural 3D-based, image-based, and video-based—and by detailing foundational representations, datasets, evaluations, and applications. It provides a structured taxonomy, highlights the trade-offs between realism, control, and efficiency, and surveys both traditional and emerging techniques from LLM-guided procedural systems to diffusion-enabled NeRFs and beyond. The article also synthesize datasets, evaluation protocols, and practical applications in editing, embodied AI, robotics, and autonomous driving, while outlining key challenges such as generative capacity, data availability, and standardized benchmarks. Finally, it offers future directions toward higher fidelity, physics-aware and interactive generation, and unified perception-generation models to advance 3D scene synthesis for real-world deployment.

Abstract

3D scene generation seeks to synthesize spatially structured, semantically meaningful, and photorealistic environments for applications such as immersive media, robotics, autonomous driving, and embodied AI. Early methods based on procedural rules offered scalability but limited diversity. Recent advances in deep generative models (e.g., GANs, diffusion models) and 3D representations (e.g., NeRF, 3D Gaussians) have enabled the learning of real-world scene distributions, improving fidelity, diversity, and view consistency. Recent advances like diffusion models bridge 3D scene synthesis and photorealism by reframing generation as image or video synthesis problems. This survey provides a systematic overview of state-of-the-art approaches, organizing them into four paradigms: procedural generation, neural 3D-based generation, image-based generation, and video-based generation. We analyze their technical foundations, trade-offs, and representative results, and review commonly used datasets, evaluation protocols, and downstream applications. We conclude by discussing key challenges in generation capacity, 3D representation, data and annotations, and evaluation, and outline promising directions including higher fidelity, physics-aware and interactive generation, and unified perception-generation models. This review organizes recent advances in 3D scene generation and highlights promising directions at the intersection of generative AI, 3D vision, and embodied intelligence. To track ongoing developments, we maintain an up-to-date project page: https://github.com/hzxie/Awesome-3D-Scene-Generation.

Figures (6)

• Figure 1: Annual statistics of 3D scene generation papers in computer vision conferences, journals, and preprints. The notable rise in publications and the evolving trends in recent years highlight the need for a comprehensive survey. Note that the data for 2025 reflects papers published up until April 30th.
• Figure 2: The overall structure of our comprehensive survey. Our survey presents three core contributions: 1) a summary of key representations and generative models in 3D scene generation, 2) a hierarchical taxonomy systematically organizing intertwined papers with in-depth analysis, and 3) an exploration of datasets, evaluation metrics, applications, along with an outlook on challenges and future directions.
• Figure 3: The paradigms of procedural methods for 3D scene generation.(a) Rule-based generation methods follow predefined rules to generate 3D scenes. (b) Optimization-based generation finds an optimized scene under different constraints. (c) LLM-based generation uses large language models (LLMs) for tasks like layout design and object selection, or to generate code that controls other generators. Note that dashed arrows denote optional operations. "Optm.", "Ret.", and "SG." denote "Optimization", "Retrieval", and "Shape Generation", respectively. "Interaction" refers to user actions such as click, drag, or selection during the generation process.
• Figure 4: The paradigms of neural 3D-based methods for 3D scene generation. These paradigms use (a) scene parameters, (b) scene graphs, (c) semantic layouts, and (d) implicit layouts as intermediate representations to control the spatial arrangement of generated 3D scenes. These representations, either user-provided or produced by generative models, are then converted into 3D scene representations (e.g., voxel grid, mesh, NeRF, or 3D Gaussians) via retrieval or decoding. Note that dashed arrows denote optional operations. "Enc." and "Dec." stand for "Encoder" and "Decoder", respectively. "Shape Gen." represents "Shape Generation".
• Figure 5: The paradigms of image-based methods for 3D scene generation.(a) Holistic generation creates an entire scene image in one step. (b) Iterative generation progressively extends the scene by extrapolating a sequence of images.