Pro-DG: Procedural Diffusion Guidance for Architectural Facade Generation

TL;DR

Pro-DG tackles the challenge of editing architectural facades by tightly coupling a neuro-symbolic procedural grammar with diffusion-based synthesis. It reconstructs a facade’s procedural representation from an input image, applies user-driven structural edits, and guides the diffusion process through hierarchical symbol matching and controlled conditioning. The approach introduces a novel SVD-based structural similarity and a content-aware histogram metric to robustly align original and edited structures, enabling faithful, globally coherent edits while preserving architectural identity. Quantitative and qualitative evaluations, complemented by a user study, demonstrate improved edit adherence and identity preservation compared with baselines, highlighting the practical impact of integrating symbolic grammars with modern generative models for structured image editing.

Abstract

We present Pro-DG, a framework for procedurally controllable photo-realistic facade generation that combines a procedural shape grammar with diffusion-based image synthesis. Starting from a single input image, we reconstruct its facade layout using grammar rules, then edit that structure through user-defined transformations. As facades are inherently multi-hierarchical structures, we introduce hierarchical matching procedure that aligns facade structures at different levels which is used to introduce control maps to guide a generative diffusion pipeline. This approach retains local appearance fidelity while accommodating large-scale edits such as floor duplication or window rearrangement. We provide a thorough evaluation, comparing Pro-DG against inpainting-based baselines and synthetic ground truths. Our user study and quantitative measurements indicate improved preservation of architectural identity and higher edit accuracy. Our novel method is the first to integrate neuro-symbolically derived shape-grammars for modeling with modern generative model and highlights the broader potential of such approaches for precise and controllable image manipulation.

Figures (13)

• Figure 1: Pro-DG is a novel approach to guiding diffusion model outputs by using procedural definitions to control the generation process. The method is able to generate new facade variations based on the provided target procedural structure.
• Figure 2: The pipeline consists of two distinct elements: the Hierarchical Matching Pipeline & the Guided Inference Pipeline. The first one is responsible for finding the correspondences between the procedural representations while the second one guides the diffusion process based on those correspondences.
• Figure 3: An example of a simplified procedural representation using a split grammar derivation tree. The tree defines the hierarchical structure of a facade and covers the whole image space.
• Figure 4: Example of the fully reconstructed Canny edges serve as guidance during the inference process. The new Canny edges image is created by transforming the original Canny edges image according to the hierarchical pairings.
• Figure 5: Visualization of the Photoshop inpainting baseline creation process. A collage of facade elements is assembled, followed by inpainting to complete the remaining portions of the facade.