Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Spatial Blindspot of Vision-Language Models

Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A, Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna

TL;DR

The paper tackles the lack of spatial grounding in vision–language models caused by flattening 2D images into patch sequences. It evaluates alternative image encoders (SigLIP, SigLIP2, AIMv2) within the LLaVA framework and introduces 2D-RoPE to preserve 2D structure during vision–language alignment. Across general and spatial benchmarks, encoder choice and 2D positional encoding significantly shape spatial reasoning, with frontier models like Qwen2-VL often performing best and AIMv2 delivering strong, consistent spatial improvements in several tasks. The findings highlight encoder design as a key driver of spatial awareness in VLMs and point to future work extending to 3D, dynamic scenes, and alternative alignment strategies to further close the gap in spatial understanding for embodied AI applications.

Abstract

Vision-language models (VLMs) have advanced rapidly, but their ability to capture spatial relationships remains a blindspot. Current VLMs are typically built with contrastive language-image pretraining (CLIP) style image encoders. The training recipe often flattens images into 1D patch sequences, discarding the 2D structure necessary for spatial reasoning. We argue that this lack of spatial awareness is a missing dimension in VLM design and a bottleneck for applications requiring spatial grounding, such as robotics and embodied AI. To address this, we investigate (i) image encoders trained with alternative objectives and (ii) 2D positional encodings. Our experiments show that these architectural choices can lead to improved spatial reasoning on several benchmarks.

The Spatial Blindspot of Vision-Language Models

TL;DR

The paper tackles the lack of spatial grounding in vision–language models caused by flattening 2D images into patch sequences. It evaluates alternative image encoders (SigLIP, SigLIP2, AIMv2) within the LLaVA framework and introduces 2D-RoPE to preserve 2D structure during vision–language alignment. Across general and spatial benchmarks, encoder choice and 2D positional encoding significantly shape spatial reasoning, with frontier models like Qwen2-VL often performing best and AIMv2 delivering strong, consistent spatial improvements in several tasks. The findings highlight encoder design as a key driver of spatial awareness in VLMs and point to future work extending to 3D, dynamic scenes, and alternative alignment strategies to further close the gap in spatial understanding for embodied AI applications.

Abstract

Vision-language models (VLMs) have advanced rapidly, but their ability to capture spatial relationships remains a blindspot. Current VLMs are typically built with contrastive language-image pretraining (CLIP) style image encoders. The training recipe often flattens images into 1D patch sequences, discarding the 2D structure necessary for spatial reasoning. We argue that this lack of spatial awareness is a missing dimension in VLM design and a bottleneck for applications requiring spatial grounding, such as robotics and embodied AI. To address this, we investigate (i) image encoders trained with alternative objectives and (ii) 2D positional encodings. Our experiments show that these architectural choices can lead to improved spatial reasoning on several benchmarks.
Paper Structure (15 sections, 1 equation, 4 figures, 5 tables)

This paper contains 15 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Encoders
  4. Spatial Reasoning
  5. VLMs and Spatial Awareness
  6. Measuring Spatial Awareness
  7. Experimental Setup
  8. Methods
  9. Training Strategies
  10. Results
  11. Evaluating on General Purpose Benchmarks
  12. Evaluating on Spatial Benchmarks
  13. Conclusion
  14. Future Work
  15. Acknowledgment

Figures (4)

  • Figure 1: Our experimental approach with LLaVA Framework liu2023llava that compares the performance of different image encoders and 2D-RoPE variants.
  • Figure 2: Pretraining strategies with various image backbones and 2D-RoPE
  • Figure 3: Example image from LLaVA-Bench (In-the-Wild) liu2023llava.
  • Figure 4: Object localization in LLaVA-SigLIP2 vs. LLaVA-AIMv2 for different prompts.