BuildingBlock: A Hybrid Approach for Structured Building Generation

TL;DR

BuildingBlock tackles the problem of generating diverse, hierarchically coherent 3D buildings from text prompts by coupling a Transformer-based diffusion model for box-based layouts with LLM-driven rule-based layout extension, followed by PCG-driven construction. The two-phase workflow—Layout Generation Phase and Building Construction Phase—enables global structural control and localized editing, producing high-quality, editable, and highly structured buildings. A new 3D architectural layout dataset (1.2k buildings, 42k boxes, 9.6k rendered images) supports training and evaluation, with additional validation on indoor scenes, where the approach achieves state-of-the-art results on multiple benchmarks. The method’s strong editing capabilities and generalizable design offer a scalable, intuitive workflow for architectural generation and potentially other controllable structured tasks.

Abstract

Three-dimensional building generation is vital for applications in gaming, virtual reality, and digital twins, yet current methods face challenges in producing diverse, structured, and hierarchically coherent buildings. We propose BuildingBlock, a hybrid approach that integrates generative models, procedural content generation (PCG), and large language models (LLMs) to address these limitations. Specifically, our method introduces a two-phase pipeline: the Layout Generation Phase (LGP) and the Building Construction Phase (BCP). LGP reframes box-based layout generation as a point-cloud generation task, utilizing a newly constructed architectural dataset and a Transformer-based diffusion model to create globally consistent layouts. With LLMs, these layouts are extended into rule-based hierarchical designs, seamlessly incorporating component styles and spatial structures. The BCP leverages these layouts to guide PCG, enabling local-customizable, high-quality structured building generation. Experimental results demonstrate BuildingBlock's effectiveness in generating diverse and hierarchically structured buildings, achieving state-of-the-art results on multiple benchmarks, and paving the way for scalable and intuitive architectural workflows.

Figures (9)

• Figure 1: The pipeline of BuildingBlock. Given a text description for the building, a stylized layout profile is first generated, which is obtained by fusing two phases. A) The Layout Generation Phase has a dual-branch structure, one diffusion-based branch produces bounding box generation for the layout, and the other LLM-based branch provides the style information extractor for the components. B) Then, the Building Construction Phase employs a PCG method to generate the corresponding structured building based on the stylized layout profile. Best view on screen and zoom in.
• Figure 2: The process of box-based layout generation. A) The box-based layout representation, including categories, sizes, and locations. B) The noise addition process of the layout diffusion model. C) The denoising process of the layout diffusion model, where the traditional positional encoding is removed and replaced by spatial encoding from the spatial positions of boxes. D) The Transformer network structure used in the diffusion model.
• Figure 3: The pipeline of our PCG.
• Figure 4: Qualitative comparison on unconditional layout generation. It can be observed that the layouts generated by LayoutGPT, ATISS and DiffuScene exhibit some issues with misaligned layouts and floating components, such as windows that are not attached to walls but instead hover in the air. Our method, on the other hand, is capable of producing high-quality layouts with coordinated components. Artifacts are marked with red boxes.
• Figure 5: Qualitative comparison of buildings generated with state-of-the-art methods. Artifacts are marked with red boxes.