Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FaçAID: A Transformer Model for Neuro-Symbolic Facade Reconstruction

Aleksander Plocharski, Jan Swidzinski, Joanna Porter-Sobieraj, Przemyslaw Musialski

TL;DR

A neuro-symbolic transformer-based model that converts flat, segmented facade structures into procedural definitions using a custom-designed split grammar is introduced, providing a procedural representation that users can adjust to generate varied facade designs.

Abstract

We introduce a neuro-symbolic transformer-based model that converts flat, segmented facade structures into procedural definitions using a custom-designed split grammar. To facilitate this, we first develop a semi-complex split grammar tailored for architectural facades and then generate a dataset comprising of facades alongside their corresponding procedural representations. This dataset is used to train our transformer model to convert segmented, flat facades into the procedural language of our grammar. During inference, the model applies this learned transformation to new facade segmentations, providing a procedural representation that users can adjust to generate varied facade designs. This method not only automates the conversion of static facade images into dynamic, editable procedural formats but also enhances the design flexibility, allowing for easy modifications.

FaçAID: A Transformer Model for Neuro-Symbolic Facade Reconstruction

TL;DR

A neuro-symbolic transformer-based model that converts flat, segmented facade structures into procedural definitions using a custom-designed split grammar is introduced, providing a procedural representation that users can adjust to generate varied facade designs.

Abstract

We introduce a neuro-symbolic transformer-based model that converts flat, segmented facade structures into procedural definitions using a custom-designed split grammar. To facilitate this, we first develop a semi-complex split grammar tailored for architectural facades and then generate a dataset comprising of facades alongside their corresponding procedural representations. This dataset is used to train our transformer model to convert segmented, flat facades into the procedural language of our grammar. During inference, the model applies this learned transformation to new facade segmentations, providing a procedural representation that users can adjust to generate varied facade designs. This method not only automates the conversion of static facade images into dynamic, editable procedural formats but also enhances the design flexibility, allowing for easy modifications.
Paper Structure (34 sections, 2 equations, 10 figures, 1 table)

This paper contains 34 sections, 2 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Procedural Modeling
  4. Inverse Procedural Modeling
  5. Generative Models in Graphics
  6. Neuro-Symbolic Methods in Graphics
  7. Problem Statement and Overview
  8. Procedural Data Generation
  9. Neuro-Symbolic Generative Model
  10. Transformer-Based Inverse Procedural Learning
  11. Sequence Generation:
  12. Tokenization:
  13. Positional Encodings:
  14. Training:
  15. Neuro-Symbolic Procedure Inference
  16. ...and 19 more sections

Figures (10)

  • Figure 1: The inference and modeling pipeline of our method: (1) we start with a user provided segmented facade; (2) after preprocessing, a transformer-based model inversely extracts the procedure how to generate the facade; (3) it can be regenerated procedurally, or used for interactive generation of new variations or other facades.
  • Figure 2: Example production rules of our split grammar which follows the notion as proposed by Wonka et al. Wonka2003. The depicted productions are a subset of all productions $P$. For a full set of rules used by our grammar please refer to supplemental material.
  • Figure 3: Sizing optimizer: The input to sizing optimization is the reconstructed procedure (structural) with default sizing information. As our derived procedures are differentiable w.r.t. the sizing parameters, the terminal sizes are computed by optimizing the square error.
  • Figure 4: Quantitative evaluation of our model: The first metric shows the frequency of the usages of the procedural rules between the ground truth and reconstructions as well us their difference normalized by inference counts. The second metric measures how much tree-edits are needed to change the reconstructed rules to those from ground truth. The normalized distances represents edit distances divided by their original tree size.
  • Figure 5: Facade reconstruction error
  • ...and 5 more figures