Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA

Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A., Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna

Abstract

Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.

Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA

Abstract

Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.
Paper Structure (12 sections, 3 figures, 2 tables)

This paper contains 12 sections, 3 figures, 2 tables.

Table of Contents

  1. Background and Motivation
  2. Experimental Setup
  3. Methods
  4. Training
  5. Results
  6. Frontier models still fail on spatial reasoning
  7. Encoder choice dominates spatial performance in controlled LLaVA variants
  8. Qualitative localization reveals missing spatial grounding
  9. 2D positional structure helps, but does not resolve spatial grounding
  10. Conclusion
  11. Future Work
  12. Acknowledgment

Figures (3)

  • Figure 1: Our experimental approach with LLaVA Framework liu2023llava that compares the performance of different image encoders and 2D-RoPE variants.
  • Figure 2: Object localization in LLaVA-SigLIP2 vs. LLaVA-AIMv2. AIMv2 yields tighter and more spatially aligned localizations, while SigLIP2 often produces imprecise or misaligned boxes.
  • Figure 3: Example image from LLaVA-Bench (In-the-Wild) liu2023llava.