Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models

TL;DR

Blueprint-Bench targets the longstanding challenge of spatial reasoning by asking AI to reconstruct accurate 2D floor plans from apartment photos. It unifies LLMs, image-generation models, and agents on a single, rule-governed evaluation framework that scores connectivity and size ranking via a robust extraction-and-scoring pipeline. The striking finding is that most systems fall at or below random baselines, with humans maintaining a substantial gap, highlighting a critical blind spot in current architectures. By providing an open, numerically comparable leaderboard and accompanying code, the work establishes a baseline for measuring and driving progress in spatial intelligence across model families.

Abstract

We introduce Blueprint-Bench, a benchmark designed to evaluate spatial reasoning capabilities in AI models through the task of converting apartment photographs into accurate 2D floor plans. While the input modality (photographs) is well within the training distribution of modern multimodal models, the task of spatial reconstruction requires genuine spatial intelligence: inferring room layouts, understanding connectivity, and maintaining consistent scale. We evaluate leading language models (GPT-5, Claude 4 Opus, Gemini 2.5 Pro, Grok-4), image generation models (GPT-Image, NanoBanana), and agent systems (Codex CLI, Claude Code) on a dataset of 50 apartments with approximately 20 interior images each. Our scoring algorithm measures similarity between generated and ground-truth floor plans based on room connectivity graphs and size rankings. Results reveal a significant blind spot in current AI capabilities: most models perform at or below a random baseline, while human performance remains substantially superior. Image generation models particularly struggle with instruction following, while agent-based approaches with iterative refinement capabilities show no meaningful improvement over single-pass generation. Blueprint-Bench provides the first numerical framework for comparing spatial intelligence across different model architectures. We will continue evaluating new models as they are released and welcome community submissions, monitoring for the emergence of spatial intelligence in generalist AI systems.

Figures (13)

• Figure 1: Overview of the Blueprint-Bench task: converting apartment photographs (left) into a 2D floor plan (right). Red dots indicate rooms, and green lines show doorways of connecting rooms.
• Figure 2: Example of a geometry problem solved by Gemini 2.5 Flash Image fortin2025introducing.
• Figure 3: Three representations of floor plan analysis: (A) Traditional floor plan with labeled rooms (kitchen, bedroom, living room, bathroom, and ensuite bathroom), (B) Room connectivity graph showing adjacency relationships between rooms color-coded to match the floor plan, and (C) Rooms ordered by size from largest to smallest.
• Figure 4: Extraction algorithm output showing segmented rooms (colored regions), room IDs assigned by size rank (1=largest), detected door connections (green lines), and door locations (black circles) from a standardized floor plan image.
• Figure 5: Mean similarity scores for different models on Blueprint-Bench. Error bars show standard deviation. The horizontal black line indicates the random baseline score. Image generation models are striped and agents are dotted.