VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations

Abstract

Multi-view visual reasoning is essential for intelligent systems that must understand complex environments from sparse and discrete viewpoints, yet existing research has largely focused on single-image or temporally dense video settings. In real-world scenarios, reasoning across views requires integrating partial observations without explicit guidance, while collecting large-scale multi-view data with accurate geometric and semantic annotations remains challenging. To address this gap, we leverage physically grounded simulation to construct diverse, high-fidelity 3D scenes with precise per-view metadata, enabling scalable data generation that remains transferable to real-world settings. Based on this engine, we introduce VIEW2SPACE, a multi-dimensional benchmark for sparse multi-view reasoning, together with a scalable, disjoint training split supporting millions of grounded question-answer pairs. Using this benchmark, a comprehensive evaluation of state-of-the-art vision-language and spatial models reveals that multi-view reasoning remains largely unsolved, with most models performing only marginally above random guessing. We further investigate whether training can bridge this gap. Our proposed Grounded Chain-of-Thought with Visual Evidence substantially improves performance under moderate difficulty, and generalizes to real-world data, outperforming existing approaches in cross-dataset evaluation. We further conduct difficulty-aware scaling analyses across model size, data scale, reasoning depth, and visibility constraints, indicating that while geometric perception can benefit from scaling under sufficient visibility, deep compositional reasoning across sparse views remains a fundamental challenge.

Figures (22)

• Figure 1: A single glance is often insufficient for completing real-world tasks. Humans naturally integrate observations across sparse viewpoints to form a shared spatial understanding. Given heterogeneous views (e.g., robotic dog and drone), one can efficiently align what different agents observe and reason across sparse viewpoints.
• Figure 2: Overview of the multi-view data engine. Stage 1: Assets are collected, categorized, and size-normalized, and annotated using OSD-Tag with verification. Stage 2: Given theme configurations defining asset categories and spatial relations, scenes are synthesized by sampling layouts under compositional constraints. Stage 3: Multi-view images are rendered together with physics-based metadata. Stage 4: Spatial relations are selected to automatically generate grounded question-answer pairs.
• Figure 3: Overview of VIEW2SPACE task design. VIEW2SPACE comprises three task types: MCQ, counting, and visual grounding detection. Reasoning difficulty ranges from one-hop perceptual queries to multi-hop cross-view spatial reasoning.
• Figure 4: Scaling Behaviour across Reasoning and Key-Object Visibility Difficulty (see Sec. \ref{['def_object_vis_define']}). Each subplot plots mIoU versus task difficulty. Each curve corresponds to a specific dataset size, fitted over the averaged performance at each difficulty level (aggregated across multiple samples and weighted accordingly). The dense empirical dot plots and detailed results are provided in the appendix.
• Figure 5: Overview of VIEW2SPACE Scene Thematic Categories