SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data

TL;DR

This work tackles the lack of spatial reasoning data in vision-language models by generating a large synthetic QA dataset from hyper-detailed image captions using an LLM. The approach yields 455k samples with 3.4 million QA pairs derived from DOCCI, Localized Narratives, and PixMo-Cap, and fine-tunes VLMs to form SpaRE models. SpaRE achieves substantial gains on spatial reasoning benchmarks (up to 49% on What's Up) while maintaining strong general VL performance, demonstrating robust transfer to real-world tasks like robotics and navigation. The study emphasizes data quality and diversity as critical for spatial understanding and points to future directions in explicit frames of reference and broader multilingual support.

Abstract

Vision-language models (VLMs) work well in tasks ranging from image captioning to visual question answering (VQA), yet they struggle with spatial reasoning, a key skill for understanding our physical world that humans excel at. We find that spatial relations are generally rare in widely used VL datasets, with only a few being well represented, while most form a long tail of underrepresented relations. This gap leaves VLMs ill-equipped to handle diverse spatial relationships. To bridge it, we construct a synthetic VQA dataset focused on spatial reasoning generated from hyper-detailed image descriptions in Localized Narratives, DOCCI, and PixMo-Cap. Our dataset consists of 455k samples containing 3.4 million QA pairs. Trained on this dataset, our Spatial-Reasoning Enhanced (SpaRE) VLMs show strong improvements on spatial reasoning benchmarks, achieving up to a 49% performance gain on the What's Up benchmark, while maintaining strong results on general tasks. Our work narrows the gap between human and VLM spatial reasoning and makes VLMs more capable in real-world tasks such as robotics and navigation.

Figures (7)

• Figure 1: Our synthetic data generation pipeline: Hyper-detailed image descriptions are fed to an LLM that extracts spatial-reasoning question-answer (QA) pairs.
• Figure 2: Comparison of answers provided by different VLMs to a spatial reasoning question.
• Figure 3: Ambiguity of spatial relations without an explicit frame of reference: is the plant to the right or left of the bench from the viewer's or woman's perspective?
• Figure 4: For our qualitative analysis, each sub-figure contains an image, a corresponding question, different models’ responses, and their correctness (✔ or ✘).
• Figure 5: An example from DOCCI, one of the hyper-detailed image-captioning datasets that we extract QA pairs from. We italicize spatially relevant words for emphasis.