SARO: Space-Aware Robot System for Terrain Crossing via Vision-Language Model

TL;DR

SARO introduces a space-aware robot framework that enables 3D terrain crossing by marrying vision-language model (VLM) reasoning with a closed-loop sub-task execution and a robust low-level locomotive policy. The high-level module uses zero-shot VLM reasoning to decompose navigation into actionable sub-tasks, while the closed-loop discriminator double-checks progress to improve robustness in 3D environments. The low-level policy, trained via Probability Annealing Selection (PAS), transitions from privileged training data to proprioception-only deployment, enabling robust, terrain-adaptive locomotion across stairs, ramps, gaps, and doors. Extensive indoor and outdoor experiments, plus simulations with diverse terrains and ablations, demonstrate strong generalization, robustness, and effective sim-to-real transfer, highlighting the practical potential for VLM-driven robotic navigation in complex 3D spaces.

Abstract

The application of vision-language models (VLMs) has achieved impressive success in various robotics tasks. However, there are few explorations for these foundation models used in quadruped robot navigation through terrains in 3D environments. In this work, we introduce SARO (Space Aware Robot System for Terrain Crossing), an innovative system composed of a high-level reasoning module, a closed-loop sub-task execution module, and a low-level control policy. It enables the robot to navigate across 3D terrains and reach the goal position. For high-level reasoning and execution, we propose a novel algorithmic system taking advantage of a VLM, with a design of task decomposition and a closed-loop sub-task execution mechanism. For low-level locomotion control, we utilize the Probability Annealing Selection (PAS) method to effectively train a control policy by reinforcement learning. Numerous experiments show that our whole system can accurately and robustly navigate across several 3D terrains, and its generalization ability ensures the applications in diverse indoor and outdoor scenarios and terrains. Project page: https://saro-vlm.github.io/

Figures (9)

• Figure 1: SARO achieves space-aware navigation capability for 3D terrain crossing. Our system utilizes the reasoning and motion planning ability of the VLM model, with a design of task decomposition and a closed-loop sub-task execution module. While traditional navigation approaches (yellow) fail in reasoning 3D environment, our system (red) guides the quadruped robot to cross the accessible intermediation towards the goal.
• Figure 2: Overview of the SARO system. The robot needs to complete a goal-tracking task while autonomously navigating through a 3D terrain. The pretrained vision-language foundation model (VLM) takes as input RGB images and prompts querying the 3D environment perception to decompose the task into sub-tasks. After that, the system executes these sub-tasks in a closed-loop taking advantage of VLM discriminator double-check. The well-designed sub-task execution module connects the high-level VLM and the low-level control policy and receives depth image and stereo image information to help localization.
• Figure 3: After task decomposition, the system executes the sub-tasks one by one. The double-check closed-loop module improves the robustness of the system.
• Figure 4: Overview of the low-level locomotion control policy. In the first training step, we train an oracle policy using proprioception $\boldsymbol{p}_t\in\mathbb{R}^{45}$, privileged state $\boldsymbol{s}_t\in\mathbb{R}^{4}$, and terrain information $\boldsymbol{t}_t\in\mathbb{R}^{187}$. In the second training step, we use the probability annealing selection (PAS) method to train the final actor network, which only uses proprioception as input. After the policy training process is finished, we exclusively train a terrain estimator to classify whether the robot is on the plane or is climbing the intermediation.
• Figure 5: Robot setup for experiment.