ZeST: an LLM-based Zero-Shot Traversability Navigation for Unknown Environments

TL;DR

ZeST tackles the challenge of traversability prediction in unknown environments by using multimodal LLMs to infer region-wise traversability without physical data collection. It fuses LLM-derived samples into a probabilistic map via Normal-Inverse-Gamma modeling, applies CVaR-based risk assessment, and employs traversability-aware RRT*-based planning and MPPI control. The approach yields a global traversability map that supports zero-shot navigation with improved safety and success rates, validated in real indoor and outdoor scenarios against baselines. By combining LLM reasoning, Bayesian uncertainty modeling, and risk-aware planning, ZeST offers a scalable, data-efficient solution for robust autonomous navigation in unknown terrains.

Abstract

The advancement of robotics and autonomous navigation systems hinges on the ability to accurately predict terrain traversability. Traditional methods for generating datasets to train these prediction models often involve putting robots into potentially hazardous environments, posing risks to equipment and safety. To solve this problem, we present ZeST, a novel approach leveraging visual reasoning capabilities of Large Language Models (LLMs) to create a traversability map in real-time without exposing robots to danger. Our approach not only performs zero-shot traversability and mitigates the risks associated with real-world data collection but also accelerates the development of advanced navigation systems, offering a cost-effective and scalable solution. To support our findings, we present navigation results, in both controlled indoor and unstructured outdoor environments. As shown in the experiments, our method provides safer navigation when compared to other state-of-the-art methods, constantly reaching the final goal.

Figures (7)

• Figure 1: We present ZeST, a Zero-Shot Traversability navigation method that is able to navigate in unknown environments without the need of fine-tuning.
• Figure 2: Overview of the proposed zero-shot navigation pipeline. (1) LLM Oracle Prediction: LLM and SAM generate traversability samples. (2) Statistical Traversability Mapping: Samples are fused into a probabilistic map. (3) Risk Assessment: Navigation risk is quantified via expected shortfall. (4) Zero-Shot Navigation: The robot navigates unfamiliar environments using the risk-aware map.
• Figure 3: Example of a prompt used in our work. It gives details about the robotic platform and references for the latent traversability values.
• Figure 4: Example of traversability maps generate using ZeST. The example on top demonstrates a map generate in an indoor environment and the one at the bottom an outdoor environment. Colors towards red represent more traversable terrains.
• Figure 5: Visualization of ZeST deployed in challenging environments compared to NoMaD and CoNVOI.The top row shows the indoor experiment plotted on the traversability octomap created by ZeST. The bottom row show the outdoor experiment plotted on the top down view of the scene.