WAFFLE: Multimodal Floorplan Understanding in the Wild

TL;DR

WAFFLE introduces a scalable, multimodal floorplan dataset mined from internet sources and curated with LLMs and vision-language models to produce rich, grounded semantic annotations. It demonstrates strong utility for both discriminative tasks (building-type understanding, open-vocabulary segmentation) and generative tasks (text- and structure-conditioned floorplan generation) using diffusion-based models and ControlNet conditioning. The dataset comprises roughly 20k floorplan images with diverse building types and geographies, along with ground-truth-like structured metadata and grounding of architectural features. WAFFLE enables new research directions in building understanding, robust in-the-wild floorplan analysis, and guided floorplan synthesis, contributing a foundation for future architectural understanding and related applications.

Abstract

Buildings are a central feature of human culture and are increasingly being analyzed with computational methods. However, recent works on computational building understanding have largely focused on natural imagery of buildings, neglecting the fundamental element defining a building's structure -- its floorplan. Conversely, existing works on floorplan understanding are extremely limited in scope, often focusing on floorplans of a single semantic category and region (e.g. floorplans of apartments from a single country). In this work, we introduce WAFFLE, a novel multimodal floorplan understanding dataset of nearly 20K floorplan images and metadata curated from Internet data spanning diverse building types, locations, and data formats. By using a large language model and multimodal foundation models, we curate and extract semantic information from these images and their accompanying noisy metadata. We show that WAFFLE enables progress on new building understanding tasks, both discriminative and generative, which were not feasible using prior datasets. We will publicly release WAFFLE along with our code and trained models, providing the research community with a new foundation for learning the semantics of buildings.

Figures (28)

• Figure 1: What can we understand from looking at these images? For instance, do we have a sense of what type of buildings these floorplans depict? Floorplans provide multimodal cues over the semantics and structure of buildings; however, they are often opaque for non-professionals, particularly for images lacking textual descriptions (such as the bottom images). We propose WAFFLE, a new multimodal dataset depicting floorplan images associated with rich textual descriptions. Our dataset allows for understanding in-the-wild floorplan imagery illustrating a wide array of building types. For example, a vision-and-language model finetuned on our data can correctly predict the building types for the examples depicted above (answers are provided below).
• Figure 2: Samples from WAFFLE. Above, we show images paired with their structured data, including the building name and type, country of origin, and their grounded architectural features. We also visualize the detected layout components (floorplan, legend, compass, and scale, as relevant) overlaid on top of the images.
• Figure 3: We automatically extract legends and architectural features from the image raw data (illustrated on the left, either the image metadata or OCR detections) by prompting LLMs. We associate the keys with text detected in the image, yielding grounded regions associated with semantics.
• Figure 4: Comparison of open-vocabulary segmentation probability map results. We show the input images in the first column, with the corresponding GT regions in red. $^*$Note that CC5K is a closed-vocabulary model designed for residential floorplan understanding, and therefore we cannot compare to it over additional building types (such as castles and cathedrals illustrated above). In addition to improving on the base CLIPSeg segmentation model, we outperform the strongly-supervised CC5K, suggesting that this model cannot generalize well beyond its training set distribution.
• Figure 5: Examples for generated floorplans for various building types, using the prompt "A floor plan of a <building_type>" (corresponding types are shown on top). The first row shows samples from the pretrained SD model, and the bottom three show results from the model fine-tuned on WAFFLE. As seen above, pretrained SD struggles at generating floorplans in general and often yields results that do not structurally resemble real floorplans. By contrast, our fine-tuned model can correctly generate fine-grained architectural structures, such as towers in castles or long corridors in libraries.