Scalability in Building Component Data Annotation: Enhancing Facade Material Classification with Synthetic Data

TL;DR

A Swin Transformer model is fine-tuned on a synthetic dataset generated with DALL-E and compared the performance to a similar manually annotated dataset, demonstrating a reasonable alternative to manual annotation.

Abstract

Computer vision models trained on Google Street View images can create material cadastres. However, current approaches need manually annotated datasets that are difficult to obtain and often have class imbalance. To address these challenges, this paper fine-tuned a Swin Transformer model on a synthetic dataset generated with DALL-E and compared the performance to a similar manually annotated dataset. Although manual annotation remains the gold standard, the synthetic dataset performance demonstrates a reasonable alternative. The findings will ease annotation needed to develop material cadastres, offering architects insights into opportunities for material reuse, thus contributing to the reduction of demolition waste.

Figures (6)

• Figure 1: Workflow showing all four experiments: baseline, augmented, mixed, synthetic with sub-experiments for stucco and siding.
• Figure 2: Class distributions for the different datasets.
• Figure 3: Brute force method to filter irrelevant synthetic images from accepted images.
• Figure 4: Confusion matrices for all experiments.
• Figure 5: ROC curve graphs comparing the stucco and siding sub-experiments.