SPA: 3D Spatial-Awareness Enables Effective Embodied Representation

TL;DR

This work argues that true visual intelligence for embodied AI requires 3D spatial awareness and presents SPA, a 3D-aware pretraining framework that equips a vanilla Vision Transformer with explicit spatial understanding via differentiable neural rendering on multi-view images. SPA builds a dynamic 3D feature volume, performs differentiable volumetric rendering to produce RGB-D and semantic supervision, and optimizes a multi-term loss $L_{total} = L_{render} + \lambda_{eikonal} L_{eikonal} + \lambda_{sdf} L_{sdf} + \lambda_{free} L_{free}$. Through the largest-scale embodied evaluation to date—268 tasks across 8 simulators and more than 10 baselines—SPA consistently achieves superior mean success rates and favorable rankings, demonstrating the critical role of 3D awareness in embodied performance. The authors further show strong zero-shot transfer to real-world tasks and provide extensive ablations, including a strong correlation between camera-pose accuracy and embodied performance, underscoring the practical impact of 3D-aware representations. The work also reports open-source code and weights to accelerate future research in embodied representation learning.

Abstract

In this paper, we introduce SPA, a novel representation learning framework that emphasizes the importance of 3D spatial awareness in embodied AI. Our approach leverages differentiable neural rendering on multi-view images to endow a vanilla Vision Transformer (ViT) with intrinsic spatial understanding. We present the most comprehensive evaluation of embodied representation learning to date, covering 268 tasks across 8 simulators with diverse policies in both single-task and language-conditioned multi-task scenarios. The results are compelling: SPA consistently outperforms more than 10 state-of-the-art representation methods, including those specifically designed for embodied AI, vision-centric tasks, and multi-modal applications, while using less training data. Furthermore, we conduct a series of real-world experiments to confirm its effectiveness in practical scenarios. These results highlight the critical role of 3D spatial awareness for embodied representation learning. Our strongest model takes more than 6000 GPU hours to train and we are committed to open-sourcing all code and model weights to foster future research in embodied representation learning. Project Page: https://haoyizhu.github.io/spa/.

Figures (8)

• Figure 1: Performance comparison across representations.Above: (a) Mean rank and (b) mean success rate on benchmarks. Lines represent the performance of SPA, best, and second best performance on each benchmark. Bottom: Rank distributions for 268 individual tasks, showing proportions from rank 1 to rank $\geq$ 4 counterclockwise. Our model demonstrates superior overall performance.
• Figure 2: Pipeline Overview. Given multi-view images, we randomly mask patches and input the remaining into a Vision Transformer. The upsampled latent features generate multi-view feature maps, from which we construct a feature volume to derive SDF values, SH coefficients, and semantic features. We then render depth, RGB, and semantic maps for loss computation.
• Figure 3: Overview of our large-scale embodied evaluation. We conduct the largest-scale evaluation of embodied representation learning to date. Our study encompasses 268 tasks across 8 simulators, including both single-task and language-conditioned multi-task settings. We evaluate diverse policy architectures and assess various state-of-the-art representation methods. This thorough evaluation allows us to provide a comprehensive and unbiased analysis of different representations.
• Figure 4: Correlation between mean success rate and camera pose regression error.
• Figure 5: Feature map visualization.