SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?

TL;DR

SpatiaLab addresses a core gap in vision-language spatial reasoning by introducing a large-scale, real-world benchmark with 1,400 QA pairs across six spatial categories and 30 subtypes. It evaluates a broad spectrum of models, using both MCQ and open-ended formats, and reveals a substantial human–machine gap driven by depth, occlusion, navigation, and 3D geometry challenges. The study conducts extensive analyses, including error diagnostics and ablations with SFT, chain-of-thought prompting, and multi-agent reasoning (SpatioXolver), finding that gains are uneven and format-sensitive, with open-ended generation lagging behind MCQ performance. The work emphasizes the need for geometry-aware grounding, physics-informed reasoning, and embodied data to move toward robust, human-aligned spatial understanding, and demonstrates the benchmark’s robustness and transferability across tasks and datasets.

Abstract

Spatial reasoning is a fundamental aspect of human cognition, yet it remains a major challenge for contemporary vision-language models (VLMs). Prior work largely relied on synthetic or LLM-generated environments with limited task designs and puzzle-like setups, failing to capture the real-world complexity, visual noise, and diverse spatial relationships that VLMs encounter. To address this, we introduce SpatiaLab, a comprehensive benchmark for evaluating VLMs' spatial reasoning in realistic, unconstrained contexts. SpatiaLab comprises 1,400 visual question-answer pairs across six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each with five subcategories, yielding 30 distinct task types. Each subcategory contains at least 25 questions, and each main category includes at least 200 questions, supporting both multiple-choice and open-ended evaluation. Experiments across diverse state-of-the-art VLMs, including open- and closed-source models, reasoning-focused, and specialized spatial reasoning models, reveal a substantial gap in spatial reasoning capabilities compared with humans. In the multiple-choice setup, InternVL3.5-72B achieves 54.93% accuracy versus 87.57% for humans. In the open-ended setting, all models show a performance drop of around 10-25%, with GPT-5-mini scoring highest at 40.93% versus 64.93% for humans. These results highlight key limitations in handling complex spatial relationships, depth perception, navigation, and 3D geometry. By providing a diverse, real-world evaluation framework, SpatiaLab exposes critical challenges and opportunities for advancing VLMs' spatial reasoning, offering a benchmark to guide future research toward robust, human-aligned spatial understanding. SpatiaLab is available at: https://spatialab-reasoning.github.io/.

Figures (42)

• Figure 1: Overview of SpatiaLab. The benchmark addresses limitations of prior datasets (left), introduces 1,400 visual QA pairs spanning 5 categories and 30 subcategories (center), and enables systematic evaluation through multiple-choice and open-ended tasks. It features diverse task and image complexity, with varied object counts, layers, lighting, textures, relations, and materials (right).
• Figure 2: Representative examples from six categories in open-ended and MCQ Tasks.
• Figure 3: Data creation pipeline for SpatiaLab. Images are collected via web crawling, targeted search, and manual snapshots, followed by structured annotation of spatial question–answer pairs.
• Figure 4: SFT training results: (a) learning trends over epochs, and (b) final accuracy values.
• Figure 5: Categories and subcategories of spatial reasoning in SpatiaLab. Each category decomposes into five subcategories, yielding thirty task types in total.