Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model

Youngwan Lee, Soojin Jang, Yoorhim Cho, Seunghwan Lee, Yong-Ju Lee, Sung Ju Hwang

Abstract

Spatial reasoning is foundational for Vision-Language Models (VLMs), particularly when deployed as Vision-Language-Action (VLA) agents in physical environments. However, existing benchmarks predominantly focus on elementary, single-hop relations, neglecting the multi-hop compositional reasoning and precise visual grounding essential for real-world scenarios. To address this, we introduce MultihopSpatial, offering three key contributions: (1) A comprehensive benchmark designed for multi-hop and compositional spatial reasoning, featuring 1- to 3-hop complex queries across diverse spatial perspectives. (2) Acc@50IoU, a complementary metric that simultaneously evaluates reasoning and visual grounding by requiring both answer selection and precise bounding box prediction - capabilities vital for robust VLA deployment. (3) MultihopSpatial-Train, a dedicated large-scale training corpus to foster spatial intelligence. Extensive evaluation of 37 state-of-the-art VLMs yields eight key insights, revealing that compositional spatial reasoning remains a formidable challenge. Finally, we demonstrate that reinforcement learning post-training on our corpus enhances both intrinsic VLM spatial reasoning and downstream embodied manipulation performance.

MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model

Abstract

Spatial reasoning is foundational for Vision-Language Models (VLMs), particularly when deployed as Vision-Language-Action (VLA) agents in physical environments. However, existing benchmarks predominantly focus on elementary, single-hop relations, neglecting the multi-hop compositional reasoning and precise visual grounding essential for real-world scenarios. To address this, we introduce MultihopSpatial, offering three key contributions: (1) A comprehensive benchmark designed for multi-hop and compositional spatial reasoning, featuring 1- to 3-hop complex queries across diverse spatial perspectives. (2) Acc@50IoU, a complementary metric that simultaneously evaluates reasoning and visual grounding by requiring both answer selection and precise bounding box prediction - capabilities vital for robust VLA deployment. (3) MultihopSpatial-Train, a dedicated large-scale training corpus to foster spatial intelligence. Extensive evaluation of 37 state-of-the-art VLMs yields eight key insights, revealing that compositional spatial reasoning remains a formidable challenge. Finally, we demonstrate that reinforcement learning post-training on our corpus enhances both intrinsic VLM spatial reasoning and downstream embodied manipulation performance.
Paper Structure (27 sections, 2 equations, 27 figures, 6 tables)

This paper contains 27 sections, 2 equations, 27 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MultihopSpatial
  4. Overview
  5. Dataset Construction
  6. Reinforcement Learning on MultihopSpatial-Train
  7. Experiments
  8. Experiment Setup
  9. Evaluation Metrics
  10. Main Results
  11. In-depth Analysis
  12. MultihopSpatial-Train: Utility as a Training Corpus
  13. Qualitative Results
  14. Conclusion
  15. Acknowledgments
  16. ...and 12 more sections

Figures (27)

  • Figure 1: A multi-hop spatial reasoning example for an embodied agent in the real-world scenario.
  • Figure 2: Comparison of existing benchmarks (single-hop) and MultihopSpatial benchmark. In the question text, colored spans denote the queried reasoning components: Perspective, Attribute, Position, and Relation.
  • Figure 3: Compositional structure and category definitions in MultihopSpatial.
  • Figure 4: Example MultihopSpatial questions for 2-hop and 3-hop reasoning under ego-centric (top) and exo-centric (bottom) views. We omit the phrase "and provide the bounding box coordinate of the region related to your answer" for brevity.
  • Figure 5: All model performance across hop count and ego/exo perspectives.
  • ...and 22 more figures