Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models

TL;DR

This work targets the underexplored area of spatial reasoning in vision language models by defining a four component framework (spatial relations, orientation and navigation, mental rotation, spatial visualization) and building a comprehensive benchmark that blends synthetic and real world images. It evaluates 13 state of the art VLMs across carefully isolated spatial tasks, revealing that most models perform near random and highlighting the persistent difficulty of spatial cognition in current systems. The paper also introduces a GenAI driven augmentation and provides open source code and data to spur further research. Overall, it establishes a rigorous platform for diagnosing spatial reasoning gaps and guiding future improvements in multimodal spatial understanding.

Abstract

Vision-Language Models (VLMs) have recently emerged as powerful tools, excelling in tasks that integrate visual and textual comprehension, such as image captioning, visual question answering, and image-text retrieval. However, existing benchmarks for VLMs include spatial components, which often fail to isolate spatial reasoning from related tasks such as object detection or semantic comprehension. In this paper, we address these deficiencies with a multi-faceted approach towards understanding spatial reasoning. Informed by the diverse and multi-dimensional nature of human spatial reasoning abilities, we present a detailed analysis that first delineates the core elements of spatial reasoning: spatial relations, orientation and navigation, mental rotation, and spatial visualization, and then assesses the performance of these models in both synthetic and real-world images, bridging controlled and naturalistic contexts. We analyze 13 state-of-the-art Vision-Language Models, uncovering pivotal insights into their spatial reasoning performance. Our results reveal profound shortcomings in current VLMs, with average accuracy across the 13 models approximating random chance, highlighting spatial reasoning as a persistent obstacle. This work not only exposes the pressing need to advance spatial reasoning within VLMs but also establishes a solid platform for future exploration. Code available on GitHub (https://github.com/stogiannidis/srbench) and dataset available on HuggingFace (https://huggingface.co/datasets/stogiannidis/srbench).

Figures (6)

• Figure 1: Example of VLM Responses to Mental Rotation Tasks: This example underscores a notable constraint of existing VLMs, which have difficulty in precisely understanding potential rotations of the objects shown, exposing a considerable deficiency in their spatial reasoning skills.
• Figure 2: Images from our benchmark created algorithmically, drawing inspiration from cognitive tests. Left image depicts paper folding, the middle shows the easy MRT version, and the right displays the hard variant.
• Figure 3: Examples sampled from other benchmarks are depicted as follows: Left is Maze-Navwang2025picture, the center displays EgoOrientBenchjung2024right, and on the right is Spatial-Objshiri-etal-2024-empirical.
• Figure 4: Example of a GenAI image shows a rabbit in a green meadow, challenging mental rotation skills. It asks VLMs to identify the rabbit's orientation after rotation, with "Right" as the correct choice.
• Figure 5: Comparison of Mental Rotation Images Accuracy (violet) and MRT Accuracy (teal) across various vision-language models. Models such as InternVL2.5 (26B) exhibit a significant gap between the two accuracy metrics, suggesting a stronger capability in interpreting mental rotation images compared to structured MRT tasks. In contrast, lower-performing models like LLaVa-1.5 (7B) show consistently weak performance across both categories. The overall trend indicates that while some models excel in image-based spatial reasoning, their understanding of abstract mental rotation tasks remains limited.