LaViRA: Language-Vision-Robot Actions Translation for Zero-Shot Vision Language Navigation in Continuous Environments

TL;DR

LaViRA tackles zero-shot Vision-and-Language Navigation in Continuous Environments (VLN-CE) by introducing a coarse-to-fine Language–Vision–Robot Action translation framework. By delegating high-level planning to a powerful large multimodal model, grounding to a specific visual target to a smaller grounding model, and executing with a rule-based low-level controller, it eliminates dependence on pre-trained waypoint predictors while leveraging multi-scale reasoning. The approach achieves state-of-the-art zero-shot performance on VLN-CE benchmarks and demonstrates sim-to-real transfer on two distinct robotic platforms. This modular, transparent design enhances generalization, efficiency, and practicality for real-world embodied navigation using existing foundation models. Future work will focus on robustness to ambiguous instructions and deeper domain adaptation to further close sim-to-real gaps.

Abstract

Zero-shot Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires an agent to navigate unseen environments based on natural language instructions without any prior training. Current methods face a critical trade-off: either rely on environment-specific waypoint predictors that limit scene generalization, or underutilize the reasoning capabilities of large models during navigation. We introduce LaViRA, a simple yet effective zero-shot framework that addresses this dilemma by decomposing action into a coarse-to-fine hierarchy: Language Action for high-level planning, Vision Action for perceptual grounding, and Robot Action for robust navigation. This modular decomposition allows us to leverage the distinct strengths of different scales of Multimodal Large Language Models (MLLMs) at each stage, creating a system that is powerful in its reasoning, grounding and practical control. LaViRA significantly outperforms existing state-of-the-art methods on the VLN-CE benchmark, demonstrating superior generalization capabilities in unseen environments, while maintaining transparency and efficiency for real-world deployment.

Figures (5)

• Figure 1: Prior methods rely on pre-trained waypoint prediction or value mapping with limited online planning. Our LaViRA framework instead decomposes navigation into language-level planning ("Go Front"), vision-level grounding ("Go Pixel"), and robot-level control ("Go Pose"), fully leveraging MLLMs reasoning for efficient coarse-to-fine decision-making.
• Figure 2: The LaViRA Pipeline. Our framework decomposes navigation into three sequential stages. (1) Language Action: A large MLLM processes the instruction, history, and current observation to generate a high-level plan, deciding whether to move forward, turn, backtrack, or stop. (2) Vision Action: A smaller, more efficient MLLM takes the high-level plan and progress estimation to identify a specific visual target in the chosen direction, outputting its bounding box and description. (3) Robot Action: The target's pixel coordinates are projected onto a global map, and a rule-based controller navigates the robot to the destination. This hierarchical process enables generalized and robust zero-shot vision-language navigation.
• Figure 3: Prompts for Language and Vision Actions. (Left) The prompt for the Language Action model, which takes in full context to decide on a high-level direction. (Right) The prompt for the Vision Action model, which uses the output from the first stage to ground the decision in a specific visual target.
• Figure 4: Visualization examples. (Left) Navigation visualization: The outputs of Language Action model are denoted in blue text, Vision Action model outputs bounding boxes in green and target description in red. The robot’s position and orientation are represented by a red dot with arrow, blue dots denote history waypoints, and the green cross marks the target position of the next waypoint. (Right) Failure cases visualization: Language action model misjudges direction due to ambiguous instruction; Vision action model gives correct target description but selects wrong region; Visual reconstruction errors in simulation cause incorrect target localization.
• Figure 5: Real-world experiment examples. LaViRA guides a Unitree Go1 quadruped (top) and an Agilex Cobot Magic wheeled robot (bottom) in an office. The visualization shows the third-person view of the robot's trajectory alongside the agent's ego view and targets, demonstrating successful real-world adaptation on diverse platforms.