Video-MSR: Benchmarking Multi-hop Spatial Reasoning Capabilities of MLLMs

TL;DR

Video-MSR introduces the first benchmark focused on multi-hop spatial reasoning in videos, addressing a gap where prior work emphasizes single-step perception. It defines four tasks—Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction—and builds a 3,052-video, 4,993-QA dataset via a scalable, visually-grounded pipeline with rigorous human verification. A broad evaluation of 20 MLLMs reveals that while surface perception is strong, deep MSR ability lags, with issues like spatial disorientation and hallucination in multi-step deductions; MSR-9K instruction tuning demonstrates meaningful gains (e.g., +7.82% overall on Video-MSR). The framework establishes Video-MSR as a robust foundation for future research, showing that targeted, high-quality spatial reasoning data is essential for advancing multimodal spatial intelligence.

Abstract

Spatial reasoning has emerged as a critical capability for Multimodal Large Language Models (MLLMs), drawing increasing attention and rapid advancement. However, existing benchmarks primarily focus on single-step perception-to-judgment tasks, leaving scenarios requiring complex visual-spatial logical chains significantly underexplored. To bridge this gap, we introduce Video-MSR, the first benchmark specifically designed to evaluate Multi-hop Spatial Reasoning (MSR) in dynamic video scenarios. Video-MSR systematically probes MSR capabilities through four distinct tasks: Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction. Our benchmark comprises 3,052 high-quality video instances with 4,993 question-answer pairs, constructed via a scalable, visually-grounded pipeline combining advanced model generation with rigorous human verification. Through a comprehensive evaluation of 20 state-of-the-art MLLMs, we uncover significant limitations, revealing that while models demonstrate proficiency in surface-level perception, they exhibit distinct performance drops in MSR tasks, frequently suffering from spatial disorientation and hallucination during multi-step deductions. To mitigate these shortcomings and empower models with stronger MSR capabilities, we further curate MSR-9K, a specialized instruction-tuning dataset, and fine-tune Qwen-VL, achieving a +7.82% absolute improvement on Video-MSR. Our results underscore the efficacy of multi-hop spatial instruction data and establish Video-MSR as a vital foundation for future research. The code and data will be available at https://github.com/ruiz-nju/Video-MSR.

Figures (7)

• Figure 1: Video-MSR benchmark overview. To systematically estimated the spatial reasoning capabilities of MLLMs, we introduce four distinct tasks: Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction.
• Figure 1: Data statistics of Video-MSR. The table details the number of video, QA pairs, and the difficulty distribution across different source datasets.
• Figure 2: Overview of the data construction pipeline. We integrate automated generation with dual-phase verification to produce high-quality MSR data. Starting with high-fidelity video collection, we employ Gemini-3.0-Pro as the core engine to generate template-guided QA pairs. A subsequent dual-phase filtration process, including a text-only blind test and manual expert verification, is applied to eliminate biases and hallucinations, ensuring the reliability of multi-hop spatial reasoning chains.
• Figure 3: Distribution of QA pairs across four tasks.
• Figure 4: Distribution of video lengths across different source datasets in Video-MSR.