Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang

TL;DR

It is argued that spatial intelligence can emerge inherently from 2D vision alone, rather than being imposed via explicit spatial instruction tuning, to introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation directly from unposed multi-view images.

Abstract

While Vision-Language Models (VLMs) exhibit exceptional 2D visual understanding, their ability to comprehend and reason about 3D space--a cornerstone of spatial intelligence--remains superficial. Current methodologies attempt to bridge this domain gap either by relying on explicit 3D modalities or by augmenting VLMs with partial, view-conditioned geometric priors. However, such approaches hinder scalability and ultimately burden the language model with the ill-posed task of implicitly reconstructing holistic 3D geometry from sparse cues. In this paper, we argue that spatial intelligence can emerge inherently from 2D vision alone, rather than being imposed via explicit spatial instruction tuning. To this end, we introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation directly from unposed multi-view images. Spa3R is built upon the proposed Predictive Spatial Field Modeling (PSFM) paradigm, where Spa3R learns to synthesize feature fields for arbitrary unseen views conditioned on a compact latent representation, thereby internalizing a holistic and coherent understanding of the underlying 3D scene. We further integrate the pre-trained Spa3R Encoder into existing VLMs via a lightweight adapter to form Spa3-VLM, effectively grounding language reasoning in a global spatial context. Experiments on the challenging VSI-Bench demonstrate that Spa3-VLM achieves state-of-the-art accuracy of 58.6% on 3D VQA, significantly outperforming prior methods. These results highlight PSFM as a scalable path toward advancing spatial intelligence. Code is available at https://github.com/hustvl/Spa3R.

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

TL;DR

It is argued that spatial intelligence can emerge inherently from 2D vision alone, rather than being imposed via explicit spatial instruction tuning, to introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation directly from unposed multi-view images.

Abstract

While Vision-Language Models (VLMs) exhibit exceptional 2D visual understanding, their ability to comprehend and reason about 3D space--a cornerstone of spatial intelligence--remains superficial. Current methodologies attempt to bridge this domain gap either by relying on explicit 3D modalities or by augmenting VLMs with partial, view-conditioned geometric priors. However, such approaches hinder scalability and ultimately burden the language model with the ill-posed task of implicitly reconstructing holistic 3D geometry from sparse cues. In this paper, we argue that spatial intelligence can emerge inherently from 2D vision alone, rather than being imposed via explicit spatial instruction tuning. To this end, we introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation directly from unposed multi-view images. Spa3R is built upon the proposed Predictive Spatial Field Modeling (PSFM) paradigm, where Spa3R learns to synthesize feature fields for arbitrary unseen views conditioned on a compact latent representation, thereby internalizing a holistic and coherent understanding of the underlying 3D scene. We further integrate the pre-trained Spa3R Encoder into existing VLMs via a lightweight adapter to form Spa3-VLM, effectively grounding language reasoning in a global spatial context. Experiments on the challenging VSI-Bench demonstrate that Spa3-VLM achieves state-of-the-art accuracy of 58.6% on 3D VQA, significantly outperforming prior methods. These results highlight PSFM as a scalable path toward advancing spatial intelligence. Code is available at https://github.com/hustvl/Spa3R.
Paper Structure (29 sections, 10 equations, 2 figures, 7 tables)

This paper contains 29 sections, 10 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Reconstruction and Scene Representation
  4. Vision-Language Models for 3D Reasoning
  5. Method
  6. Predictive Spatial Field Modeling
  7. Learning a Unified Spatial Representation
  8. Asymmetric View Aggregator
  9. Spa3R Encoder
  10. Spa3R Decoder
  11. Losses
  12. Grounding Reasoning in Spatial Context
  13. Experiments
  14. Setup
  15. Pre-training Datasets
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Overview of the Spa3R framework and Spa3-VLM integration.(a) The Spa3R Encoder maps unposed context views to a unified, view-invariant spatial latent representation $\boldsymbol{z}$. (b) The Spa3R Decoder synthesizes target features $\hat{\boldsymbol{F}}_t$ for arbitrary unseen views, conditioned on the spatial latent $\boldsymbol{z}$ and target camera embeddings $\boldsymbol{r}$. (c) For downstream spatial reasoning, the pre-trained Spa3R Encoder is integrated into a VLM to generate spatial representation $\boldsymbol{z}$. A lightweight Adapter fuses the VLM's native visual features $\boldsymbol{F}_V$ with spatial latent $\boldsymbol{z}$ via cross-attention, effectively grounding its reasoning in spatial context.
  • Figure 2: Qualitative visualization of learned feature fields. Our predictions exhibit a continuous and spatially coherent layout compared to the target features. (Colors are unaligned due to independent PCA projections necessitated by the cosine similarity supervision). Furthermore, Spa3R plausibly extrapolates features for occluded or unobserved regions (highlighted in red boxes), demonstrating that it has internalized a holistic 3D scene understanding rather than merely memorizing input views.