MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images

TL;DR

MonoSR tackles open-world monocular spatial reasoning by introducing a large-scale dataset that challenges models to infer 3D spatial relations from single images. The authors design a three-level task hierarchy—Foundational Perception, Perspective-Aware Imagination, and Situational Reasoning—and construct over 1 million QA samples derived from 230K monocular scenes, grounded in 3D ground-truth via scene graphs and object captions. They systematically study how auxiliary information (global scene cues, 2D visual prompts, and explicit 3D bounding boxes) affects performance, finding that explicit 3D geometry provides the largest gains and that combining 2D grounding with 3D structure yields robust improvements. A comprehensive evaluation across open- and closed-source vision-language models reveals current SR capabilities are insufficient for robust monocular, open-world reasoning, providing practical guidelines for future model design and data-centric strategies to boost monocular 3D spatial understanding.

Abstract

Spatial reasoning (SR), the ability to infer 3D spatial information from 2D inputs, is essential for real-world applications such as embodied AI and autonomous driving. However, existing research primarily focuses on indoor environments and typically relies on multi-view observations, which limits their generalizability to outdoor scenarios and constrains their applicability to monocular images, the most common real-world setting. In this work, we propose MonoSR, a large-scale monocular spatial reasoning dataset that spans diverse scenarios including indoor, outdoor, and object-centric settings, and supports multiple question types. MonoSR provides a path toward open-world monocular spatial reasoning. Beyond introducing the dataset, we evaluate advanced vision-language models to reveal their limitations on this challenging task. We further analyze whether auxiliary information is crucial for monocular spatial reasoning and offer practical guidance for designing future models. These contributions collectively establish a foundation for advancing monocular spatial reasoning in real-world, open-world environments.

Figures (17)

• Figure 1: Overview of the proposed MonoSR dataset, which spans three levels of spatial reasoning—Foundational Perception, Perspective-Aware Imagination, and Situational Reasoning—across diverse indoor, outdoor, and object-centric scenes. The dataset contains over 1 million spatial VQA samples, providing comprehensive coverage for open-world monocular spatial reasoning.
• Figure 2: Demonstration of MonoSR curation pipeline
• Figure 3: Dataset composition of MonoSR across hierarchical reasoning levels and domains. Top: Distribution of tasks across three reasoning levels. Bottom: Breakdown of question formats, including multiple-choice (MC3/MC4), numeric, and yes/no types.
• Figure 4: Auxiliary Information, which can inject more context information into VLMs from global scene information, mid-level 2D location information and more precise 3D bounding information.
• Figure 5: Benchmarking open & closed-source methods. Some unreasonable question types are filtered out depending on the scenario. Dark blue and orange indicate the best result among all models, while light colours indicate the second best result.