Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians

TL;DR

Proc-GS addresses the gap between high-fidelity 3D city rendering and scalable asset management by embedding procedural code into the 3D Gaussian Splatting framework. It decomposes buildings into base assets and a procedural code string, enabling efficient editing, cross-scene asset sharing, and infinite city generation through controlled assembly, while incorporating per-instantiation variance for realism. The MatrixBuilding dataset and the two-stage Asset Acquisition and Asset Assembly pipeline demonstrate that Procedural Code + 3D-GS can achieve competitive rendering quality with substantially reduced model size and improved robustness under sparse views, as well as superior 3D consistency in city-scale generation. The approach integrates with Houdini City Sample assets and employs building and city layout generators to assemble large, controllable virtual cities, with practical impact for games, film, autonomous driving, and embodied AI simulations that rely on photorealistic city data.

Abstract

Buildings are primary components of cities, often featuring repeated elements such as windows and doors. Traditional 3D building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. Drawing inspiration from procedural modeling techniques used in the gaming and visual effects industry, our method, Proc-GS, integrates procedural code into the 3D Gaussian Splatting (3D-GS) framework, leveraging their advantages in high-fidelity rendering and efficient asset management from both worlds. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings. This integration significantly reduces model size by utilizing shared foundational assets, enabling scalable generation with precise control over building assembly. We showcase the potential for expansive cityscape generation while maintaining high rendering fidelity and precise control on both real and synthetic cases.

Figures (12)

• Figure 1: Architectural structures in urban environments often exhibit repetitive patterns, such as the arrangement of windows and doors in buildings. Our approach focuses on extracting 3D base assets from predefined layouts and multi-view captures of buildings. Utilizing 3D-GS for reconstruction, we incorporate procedural code constraints during optimization to decompose the building entity into distinct 3D assets. (1) The top row illustrates two real-world building examples, showcasing their appearances before and after editing. (2) The bottom row demonstrates the scaled-up city-level assembly achieved using UE's City Sample assets. ProcGS not only facilitates geometry editing but also enables the generation of new buildings by modifying or defining new procedural rules, offering a flexible, efficient, and scalable framework for city assembly with high-fidelity and precise control.
• Figure 2: Example Building Procedural Code from City Sample CitySample. Different levels are distinguished by colors, each represented by a string of characters indicating the instantiation of assets: C_E (external corner), P* (pillar), and W* (window). Base assets such as C_E and W1 for Levels 1 and 2, shown on the right, are manually created by artists.
• Figure 3: Overview of ProcGS. Our pipeline consists of two stages: (1) Asset Acquisition: We acquire the base assets in the training process of the 3D-GS. These assets are then assembled according to procedural code and add variance assets to create a complete building, which is used for novel view synthesis with Gaussian Splatting. (2) Asset Assembly: We use the building generator and city layout generator to assemble these base assets into a vivid 3D city. Users provide basic urban spatial data (purple city boundary and green primary road) to city layout generator to automatically predict other roads and building layouts. The building generator then assembles base assets into complete buildings using procedural code and predicted building parameters.
• Figure 4: Bbox Adaptive Clamp. (1) Clamping Scale: if a Gaussian exceeds the soft bounding box, the scale of this Gaussian is reduced by half. (2) Clamping Position: repositions Gaussians that exceed the bounding box, realigning them within the bounding box.
• Figure 5: Qualitative results. The left section shows results from three real-world scenes, while the right section presents results from the MatrixBuilding dataset. Proc-GS achieves rendering quality comparable to 3D-GS. Green boxes in each image highlight pairs of instantiations that share the same base assets, illustrating our method's capability to effectively model variations in geometry and appearance.