From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors

TL;DR

FALCON addresses the core gap in vision-language-action models by introducing rich 3D spatial priors derived from spatial foundation models and integrating them through an Embodied Spatial Model. Spatial tokens are ingested by a dedicated Spatial-Enhanced Action Head, preserving the VLM’s language reasoning while enabling precise, geometry-aware control, with optional depth and pose cues that improve performance without retraining. Across three simulation benchmarks and eleven real-world tasks, FALCON achieves state-of-the-art results, demonstrating strong modality transferability, robustness in clutter, and effective few-shot generalization. This work offers a practical pathway to robust, spatially grounded generalist robotic policies that operate effectively with RGB-only inputs and gracefully leverage additional 3D information when available.

Abstract

Existing vision-language-action (VLA) models act in 3D real-world but are typically built on 2D encoders, leaving a spatial reasoning gap that limits generalization and adaptability. Recent 3D integration techniques for VLAs either require specialized sensors and transfer poorly across modalities, or inject weak cues that lack geometry and degrade vision-language alignment. In this work, we introduce FALCON (From Spatial to Action), a novel paradigm that injects rich 3D spatial tokens into the action head. FALCON leverages spatial foundation models to deliver strong geometric priors from RGB alone, and includes an Embodied Spatial Model that can optionally fuse depth, or pose for higher fidelity when available, without retraining or architectural changes. To preserve language reasoning, spatial tokens are consumed by a Spatial-Enhanced Action Head rather than being concatenated into the vision-language backbone. These designs enable FALCON to address limitations in spatial representation, modality transferability, and alignment. In comprehensive evaluations across three simulation benchmarks and eleven real-world tasks, our proposed FALCON achieves state-of-the-art performance, consistently surpasses competitive baselines, and remains robust under clutter, spatial-prompt conditioning, and variations in object scale and height.

Figures (15)

• Figure 1: We propose FALCON, a vision-language-action model that achieves robust 3D spatial understanding by seamless integrating spatially rich tokens and sematic features. FALCON demonstrates exceptional modality transferability by excelling with both RGB-only and multi-modal inputs, superior spatial understanding in tasks involving unseen object sizes, heights and abstract spatial instructions, and strong few-shot generalizability in real-world scenes. The model achieves state-of-the-art performance across a diverse range of benchmark evaluations.
• Figure 2: Overview of FALCON framework. FALCON integrates a 2D VLM (e.g., Kosmos-2), an Embodied Spatial Model, and a Spatial-Enhanced Action Head. At timestep $t$, the VLM processes visual observations $O_t$ and language instructions $L$ to produce a semantic action token $\hat{\mathbf{t}}_{\text{act}}$. Concurrently, the Embodied Spatial Model encodes a third-view image $I^{\text{3rd}}_t$ and optional geometric inputs into spatial tokens $\mathbf{T}_{\text{spl}}$. These are fused by the Spatial-Enhanced Action Head to generate precise robot actions $A_t$, enabling robust manipulation through joint semantic and spatial reasoning.
• Figure 3: Different modality fusion strategies between spatial and semantic action tokens.
• Figure 4: Evaluation of base tasks in cluttered scene. Base Tasks contains a total of nine distinct task suites, encompassing language grounding (cluttered scenes with random distractors) and semantic understanding (unseen object poses). Each task is evaluated over 10 different scene layouts with 10 trials, resulting in a total of 90 rollouts.
• Figure 5: Performance comparison of different methods. Simple setting (left): includeing four challenging tasks selected from Base Tasks. Unseen scenarios (right): containing three unseen variations: Unseen Object, Background, and Task Description to evaluate the robustness and generalization of all models.