SpatialMosaic: A Multiview VLM Dataset for Partial Visibility

TL;DR

SpatialMosaic tackles the challenge of 3D spatial reasoning under partial visibility, occlusion, and low-overlap by introducing a scalable data-generation pipeline that yields 2M instruction-tuning QA pairs and a 1M QA benchmark across 6 tasks. It couples a geometry-aware encoder (VGGT) with a vision-language backbone to build SpatialMosaicVLM, fusing per-view visual tokens with geometry tokens through cross-attention to reason about fragmented multi-view cues. Empirical results show that instruction-tuning on SpatialMosaic improves robustness to incomplete visual evidence and yields strong zero-shot generalization to unseen multi-view spatial tasks, outperforming several baselines including larger models on SpatialMosaic-Bench and VSTI-Bench. This work advances scalable, real-world applicable multi-view reasoning by leveraging explicit 3D geometry within VLMs, enabling more reliable 3D scene understanding from limited viewpoints.

Abstract

The rapid progress of Multimodal Large Language Models (MLLMs) has unlocked the potential for enhanced 3D scene understanding and spatial reasoning. However, existing approaches often rely on pre-constructed 3D representations or off-the-shelf reconstruction pipelines, which constrain scalability and real-world applicability. A recent line of work explores learning spatial reasoning directly from multi-view images, enabling Vision-Language Models (VLMs) to understand 3D scenes without explicit 3D reconstructions. Nevertheless, key challenges that frequently arise in real-world environments, such as partial visibility, occlusion, and low-overlap conditions that require spatial reasoning from fragmented visual cues, remain under-explored. To address these limitations, we propose a scalable multi-view data generation and annotation pipeline that constructs realistic spatial reasoning QAs, resulting in SpatialMosaic, a comprehensive instruction-tuning dataset featuring 2M QA pairs. We further introduce SpatialMosaic-Bench, a challenging benchmark for evaluating multi-view spatial reasoning under realistic and challenging scenarios, consisting of 1M QA pairs across 6 tasks. In addition, we present SpatialMosaicVLM, a hybrid framework that integrates 3D reconstruction models as geometry encoders within VLMs for robust spatial reasoning. Extensive experiments demonstrate that our proposed dataset and VQA tasks effectively enhance spatial reasoning under challenging multi-view conditions, validating the effectiveness of our data generation pipeline in constructing realistic and diverse QA pairs. Code and dataset will be available soon.

Figures (24)

• Figure 1: We present SpatialMosaic, a benchmark designed to evaluate 3D spatial reasoning capabilities from fragmented visual cues across multiple viewpoints. Our benchmark focuses on three challenging real-world scenarios involving partial visibility, occlusion, and low-overlap, where current MLLMs often struggle to maintain geometric and cross-view consistency.
• Figure 2: SpatialMosaic data generation pipeline. Given multi-view images dataset, with 3D annotations (Sec. \ref{['subsec:data_preparation']}), we compute object-level and image-level occlusion ratios for each instance. Images are then filtered by overlap to ensure diverse viewpoints, and instances are filtered based on visibility constraints (Sec. \ref{['subsec:QA_Generation_and_Relations']}). Finally, spatial relations are computed and used to populate task-specific templates, generating geometrically grounded QA pairs (Sec. \ref{['subsec:QA Template and Output']}).
• Figure 3: Occlusion ratio calculation. We render each instance independently to measure visible (green) and occluded (magenta) pixels. Object Occlusion: Object-level occlusion ($r_{\text{obj}}$) captures inter-object obstruction from the actual camera view. Field-of-view Occlusion: Field-of-view truncation ($r_{\text{FoV}}$) uses extended field-of-view rendering to quantify boundary occlusion from frame cropping.
• Figure 4: SpatialMosaicVLM architecture. Multi-image inputs are processed through parallel Geometry and Visual Encoders to extract 3D structural and appearance features. The resulting geometry and visual tokens are fused via cross-attention, then combined with question tokens and processed by a Large Language Model to answer spatial reasoning questions under occlusion and partial visibility circumstances.
• Figure 6: Comparison between SpatialMosaicVLM and InternVL2-8B on SpatialMosaic-Bench.