VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation

TL;DR

Vamos presents a hierarchical VLA navigation framework that separates general planning from embodiment grounding to enable cross-embodiment and steerable navigation. A high-level vision-language model planner operates on diverse real-world data to predict 2D pixel-space trajectories, while a lightweight, embodiment-specific affordance model trained in simulation re-ranks and grounds these plans for actual robots. The key contributions include the 2D path interface, affordance-conditioned modulation (F_pi and F^c), and the demonstrating of cross-embodiment transfer, natural-language steerability, and significant reliability gains (e.g., 3× higher success by rejecting infeasible plans). Empirical results show Vamos outperforming state-of-the-art baselines on indoor and outdoor courses and achieving robust cross-embodiment navigation between legged and wheeled platforms, with strong sim-to-real transfer validated through extensive ablations.

Abstract

A fundamental challenge in robot navigation lies in learning policies that generalize across diverse environments while conforming to the unique physical constraints and capabilities of a specific embodiment (e.g., quadrupeds can walk up stairs, but rovers cannot). We propose VAMOS, a hierarchical VLA that decouples semantic planning from embodiment grounding: a generalist planner learns from diverse, open-world data, while a specialist affordance model learns the robot's physical constraints and capabilities in safe, low-cost simulation. We enabled this separation by carefully designing an interface that lets a high-level planner propose candidate paths directly in image space that the affordance model then evaluates and re-ranks. Our real-world experiments show that VAMOS achieves higher success rates in both indoor and complex outdoor navigation than state-of-the-art model-based and end-to-end learning methods. We also show that our hierarchical design enables cross-embodied navigation across legged and wheeled robots and is easily steerable using natural language. Real-world ablations confirm that the specialist model is key to embodiment grounding, enabling a single high-level planner to be deployed across physically distinct wheeled and legged robots. Finally, this model significantly enhances single-robot reliability, achieving 3X higher success rates by rejecting physically infeasible plans. Website: https://vamos-vla.github.io/

Figures (17)

• Figure 1: We present Vamos, a general-purpose hierarchical VLA for navigation. Our key idea is to decouple semantic planning from embodiment grounding. We achieve this by training a high-level VLM planner with diverse, heterogeneous real-world data that proposes trajectory candidates as 2D paths, which are then re-ranked by an embodiment-specific affordance model trained cheaply and safely in simulation. This yields robust, cross-embodied and steerable open-world navigation controllers.
• Figure 2: The high-level planner is a VLM trained to take an image and a goal coordinate (encoded as text) as input, optionally appending natural language preferences, and to output a set of candidate paths in pixel space. These paths are encoded as strings of location token pairs, then decoded and projected from 2D pixel space to the 3D ground plane. Finally, a capability-aware affordance function evaluates and re-ranks the 3D candidate paths to determine which path the robot should execute in the real world based on low-level policy capabilities.
• Figure 3: We fine-tune a VLM with navigation-specific, real-world datasets, with heterogeneous embodiments and capabilities, to obtain a general-puropse high-level planner. We use a filtered data mix from SCAND karnan2022socially, TartanDrive 2 sivaprakasam2024tartandrive, CODa zhang2024towards, and a small, 0.3 hour in-domain dataset collected on Spot.
• Figure 4: We run real-world navigation experiments indoors and outdoors in unseen scenes with challenging terrain, lighting, and vegetation. Our results show that Vamos outperforms state-of-the-art navigation foundation models and model-based baselines.
• Figure 5: Top-down map showing paths taken by different methods from start (red) to goal (green) through waypoints (yellow). Vamos achieves long-horizon, precise navigation. Right: predicted and selected paths when replanning after reaching a waypoint. Dotted lines show returns to the last completed waypoint after interventions; X's mark baseline failures or timeouts.