Pyramid Diffusion for Fine 3D Large Scene Generation

TL;DR

This work tackles the challenge of generating large-scale 3D outdoor scenes with diffusion models, constrained by data availability and memory. It introduces Pyramid Discrete Diffusion (PDD), a coarse-to-fine, multi-scale diffusion framework with scene subdivision and scale-adaptive conditioning to progressively refine scenes from small to large scales, enabling both unconditional and conditional generation and cross-dataset transfer. The approach demonstrates superior semantic coherence and fidelity compared with strong baselines, and extends to infinite scene generation via subdivision-guided refinement, while preserving training efficiency through parallelizable scale-specific models. The results indicate that PDD can efficiently adapt synthetic-trained models to real-world data and scale to unbounded scenes, offering practical impact for autonomous driving, robotics, and virtual environment creation.

Abstract

Diffusion models have shown remarkable results in generating 2D images and small-scale 3D objects. However, their application to the synthesis of large-scale 3D scenes has been rarely explored. This is mainly due to the inherent complexity and bulky size of 3D scenery data, particularly outdoor scenes, and the limited availability of comprehensive real-world datasets, which makes training a stable scene diffusion model challenging. In this work, we explore how to effectively generate large-scale 3D scenes using the coarse-to-fine paradigm. We introduce a framework, the Pyramid Discrete Diffusion model (PDD), which employs scale-varied diffusion models to progressively generate high-quality outdoor scenes. Experimental results of PDD demonstrate our successful exploration in generating 3D scenes both unconditionally and conditionally. We further showcase the data compatibility of the PDD model, due to its multi-scale architecture: a PDD model trained on one dataset can be easily fine-tuned with another dataset. Code is available at https://github.com/yuhengliu02/pyramid-discrete-diffusion.

Figures (19)

• Figure 1: We present Pyramid Discrete Diffusion Model, a method that progresses from generating coarse- to fine-grained scenes, mirroring the top-down sequence of the pyramid structure shown. The model is extended for cross-dataset and infinite scene generation, with detailed scene intricacies illustrated on the flanking sides of the image. $\mathcal{D}_s$ and $\mathcal{D}_t$ refer to a source dataset and a target dataset, respectively.
• Figure 2: Framework of the proposed Pyramid Discrete Diffusion model. In our structure, there are three different scales. Scenes generated by a previous scale can serve as a condition for the current scale after processing through our scale adaptive function. Furthermore, for the final scale processing, the scene from the previous scale is subdivided into four sub-scenes. The final scene is reconstructed into a large scene using our Scene Subdivision module.
• Figure 2: Different scales in the 3D scene pyramid.
• Figure 3: Visualization of unconditional generation results on CarlaSC. We compare with two baseline models -- DiscreteDiff austin2021structured and LatentDiff lee2023diffusion and show synthesis from our models with different scales. Our method produces more diverse scenes compared to the baseline models. Furthermore, with more levels, our model can synthesize scenes with more intricate details.
• Figure 4: Data retrieval visualization. We generate 1$k$ scenes using PDD on CarlaSC dataset, and retrieve the most similar scene in training set for each scene using SSIM (SSIM=1 means identical), and plot SSIM distribution. Scenes at various percentiles are displayed (red box: generated scenes; grey box: scenes in training set), those with the highest 10% similarity (i.e., 10$^{th}$ percentile of SSIM) are very similar to the training set, but still not completely identical.