Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

TL;DR

This paper introduces SpatialEval, a four-task benchmark designed to probe spatial reasoning in both LLMs and VLMs under text-only, vision-only, and vision-text modalities. Through extensive evaluation across open-source and proprietary models, it uncovers counterintuitive findings: spatial reasoning remains hard for many models, visual inputs alone often underperform, and textual information drives performance; moreover, incorporating redundant textual descriptions with visuals can significantly improve results. The authors argue that current VLM architectures primarily translate vision into language space, which may cap spatial capabilities, and they advocate for future designs where vision serves as a first-class, joint reasoning modality. Overall, SpatialEval provides a diagnostic framework that can guide the development of multimodal models with robust spatial intelligence and closer alignment to human spatial reasoning.

Abstract

Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning -- a fundamental component of human cognition -- remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence.

Figures (16)

• Figure 1: Illustration of the Spatial-Map task, which simulates a map with multiple locations. To investigate the impact of modality, we consider three input formats: Text-only, Vision-only, and Vision-text. We evaluate language models (w. TQA input) and vision-language models (w. VQA and VTQA inputs) on the same set of questions.
• Figure 2: Illustration of the Maze-Nav task, which evaluates the model's ability to navigate from the starting point (S) to the exit (E).
• Figure 3: Illustration of the Spatial-Grid task, which evaluates the model's spatial reasoning ability in a rigid grid structure.
• Figure 4: Illustration of the Spatial-Real task, which is built on real images with long captions, featuring detailed descriptions averaging over 1,000 words per image.
• Figure 5: Performance overview on spatial reasoning tasks. We report the accuracy averaged over all questions. We consider the VQA (Vision-only) format for vision-language models. The dashed red line denotes the expected accuracy for random guessing. For Spatial-Map and Maze-Nav tasks, only a few models outperform random guessing by a notable margin.