Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

TL;DR

Urban Architect addresses the challenge of scaling text-to-3D generation to city-scale scenes by introducing a compositional 3D layout prior that complements textual prompts. The method LG-VSD injects layout constraints into diffusion-based score distillation via rendered 2D semantic/depth maps and a ControlNet framework, while a Scalable Hash Grid enables unbounded urban scenes. A layout-aware refinement and explicit editing primitives yield high-quality, steerable, large-scale 3D urban scenes, demonstrated on KITTI-360 with >1000 m driving distance. The work advances urban simulation and VR/Autonomous-driving applications by providing controllable, scalable 3D city models.

Abstract

Text-to-3D generation has achieved remarkable success via large-scale text-to-image diffusion models. Nevertheless, there is no paradigm for scaling up the methodology to urban scale. Urban scenes, characterized by numerous elements, intricate arrangement relationships, and vast scale, present a formidable barrier to the interpretability of ambiguous textual descriptions for effective model optimization. In this work, we surmount the limitations by introducing a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships, complementing textual descriptions and enabling steerable generation. Upon this, we propose two modifications -- (1) We introduce Layout-Guided Variational Score Distillation to address model optimization inadequacies. It conditions the score distillation sampling process with geometric and semantic constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes, we represent 3D scene with a Scalable Hash Grid structure, incrementally adapting to the growing scale of urban scenes. Extensive experiments substantiate the capability of our framework to scale text-to-3D generation to large-scale urban scenes that cover over 1000m driving distance for the first time. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation. Website: https://urbanarchitect.github.io.

Figures (17)

• Figure 1: Urban Architect for steerable 3D urban scene generation. We present Urban Architect, a method to generate steerable 3D urban scenes by introducing 3D layout as an additional prior to complement textual descriptions. The framework enjoys three key properties: $(1)$ Large-scale urban scene creation. The scale goes beyond $1000$m driving distance in our experiments. $(2)$ High quality. The generated scene enables photo-realistic rendering (upper row) and obeys geometric consistency (the left side of the first camera's upper image). $(3)$ Steerable creation process. It supports various scene editing effects by fine-tuning the generated scene in a breeze (e.g., style editing (lower row)). Project page: https://urbanarchitect.github.io/.
• Figure 2: Overview of Urban Architect. We introduce Urban Architect, a method that generates urban-scale 3D scenes with 3D layout instruction and textural descriptions. The scene is represented by a neural field that is optimized by distilling a pre-trained diffusion model in a conditional manner. (a) Rather than relying solely on the text-based guidance, we propose to control the distilling process of Variational Score Distillation (VSD) via the 3D layout of the desired scene, introducing Layout-Guided Variational Score Distillation (LG-VSD). (b) We refine the local details via a layout-aware refinement strategy. (c) To model unbounded urban scenes, we discretize the 3D representation into a scalable hash grid. (d) We support various scene editing effects by fine-tuning the generated scene.
• Figure 3: Illustration of the scalable hash grid representation. We decomposed the scene into a set of stuff and object hash grids (i.e., $\{\mathcal{H}_k^s, \mathcal{H}_k^o\}$). The grids grow with the camera trajectory in a dynamic manner.
• Figure 4: Basic primitives. We provide several basic primitives of common objects in urban scenes (e.g., road and sidewalk, car, building, etc.)
• Figure 5: Automatic Layout Generation. We present an alternative method for automatic 3D layout generation given a single example of the layout. In the top two rows, We display generated layouts with different scales and provide the corresponding 3D layout given the generated 2D sample in the third row. The rendering results of the generated scene are displayed in the bottom row.