From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs

TL;DR

OpenBench establishes an open-world, metrically grounded benchmark for spatial intelligence in Multimodal LLMs, using pedestrian-perspective multi-sensor data to automatically generate QA across relational, metric, and kinematic tasks. The study reveals that current MLLMs rely heavily on linguistic priors, fail to transfer indoor spatial gains to outdoor settings, and struggle with dynamic spatial reasoning. It also shows that improving raw visual inputs or model scale alone does not close the spatial reasoning gap, highlighting the need for explicit 3D representations and temporally coherent world models. These findings provide a principled platform to diagnose spatial grounding deficiencies and guide future directions toward physically grounded, geometry-aware AI systems.

Abstract

While Multimodal Large Language Models (MLLMs) have achieved impressive performance on semantic tasks, their spatial intelligence--crucial for robust and grounded AI systems--remains underdeveloped. Existing benchmarks fall short of diagnosing this limitation: they either focus on overly simplified qualitative reasoning or rely on domain-specific indoor data, constrained by the lack of outdoor datasets with verifiable metric ground truth. To bridge this gap, we introduce a large-scale benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors. This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions that span a hierarchical spectrum--from qualitative relational reasoning to quantitative metric and kinematic understanding. Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings. Further analysis using synthetic abnormal scenes and blinding tests confirms that current MLLMs depend heavily on linguistic priors instead of grounded visual reasoning. Our benchmark thus provides a principled platform for diagnosing these limitations and advancing physically grounded spatial intelligence.

Figures (19)

• Figure 1: Tasks and composition of our OpenBench.(Left) Representative question-answer examples for our three tiers. (Right) Illustration of the final task distribution, which is balanced across both tiers and tasks.
• Figure 2: Overview of our benchmark construction pipeline.
• Figure 3: Evaluation results for MLLMs. We highlight the best and second-best results for each sub-task in deeper gray and light gray.
• Figure 4: Performance comparison on VSI-Bench and OpenBench across different sizes and model versions. We evaluated models from two families, QwenVL and InternVL, each with an older and a newer version. All experiments used inputs of 32 frames for consistency. (a)(b): the overall score of the models across various sizes on the indoor VSI-Bench and OpenBench; lighter colors denote older model versions; the green arrow indicates the performance gain of InternVL3.5-38B over InternVL2-40B. (c): the MRA change on the absolute distance task when comparing InternVL3.5 to InternVL2. The green line highlights the performance gain on VSI-Bench, while the red line shows the performance drop on OpenBench. † For plotting purposes, models are grouped by their approximate parameter scale. Refer to Sec. F.4 in the supplementary material for detailed results.
• Figure 5: Illustrations and results of the synthetic test set. The bar chart (left) shows the performance ($\mathcal{MRA}$) drop of humans and Gemini-2.5-Pro on the Size and Distance tasks when evaluated on abnormal scenes versus normal scenes.