Navigation with VLM framework: Towards Going to Any Language

TL;DR

NavVLM addresses open-goal navigation in unknown indoor environments by extending ObjNav to non-object and open-language goals. It uses a training-free framework that leverages a small open-source Vision-Language Model as a cognitive core to perceive, reason, and guide navigation via text prompts, with a short-term target area, fast path planning, and a backup strategy. In simulation, NavVLM achieves state-of-the-art SPL on MP3D, HM3D, and Gibson, and demonstrates open-goal language navigation episodes, plus real-world validation confirms practical viability on indoor scenes. The approach highlights the potential of VLMs to drive robust navigation without extensive environment priors or training, enabling flexible, human-friendly navigation.

Abstract

Navigating towards fully open language goals and exploring open scenes in an intelligent way have always raised significant challenges. Recently, Vision Language Models (VLMs) have demonstrated remarkable capabilities to reason with both language and visual data. Although many works have focused on leveraging VLMs for navigation in open scenes, they often require high computational cost, rely on object-centric approaches, or depend on environmental priors in detailed human instructions. We introduce Navigation with VLM (NavVLM), a training-free framework that harnesses open-source VLMs to enable robots to navigate effectively, even for human-friendly language goal such as abstract places, actions, or specific objects in open scenes. NavVLM leverages the VLM as its cognitive core to perceive environmental information and constantly provides exploration guidance achieving intelligent navigation with only a neat target rather than a detailed instruction with environment prior. We evaluated and validated NavVLM in both simulation and real-world experiments. In simulation, our framework achieves state-of-the-art performance in Success weighted by Path Length (SPL) on object-specifc tasks in richly detailed environments from Matterport 3D (MP3D), Habitat Matterport 3D (HM3D) and Gibson. With navigation episode reported, NavVLM demonstrates the capabilities to navigate towards any open-set languages. In real-world validation, we validated our framework's effectiveness in real-world robot at indoor scene.

Figures (9)

• Figure 1: (a) NavVLM can perform intelligent exploration in open scenes for object-centric goals. (b) NavVLM can guide to out-of-domain language goals. (c) NavVLM can navigate to any human-friendly open goals such as action, or even abstract places.
• Figure 2: The overall framework. At each step, framework use textual prompts to provide navigation guidance based on the current observations. After projection of text guidance, the robot will go to the target area given by VLM. In this process, the VLM perceives information and acts as a high level intelligent commander, taking control of navigation.
• Figure 3: VLM text to short-term goal
• Figure 4: Path plan overview
• Figure 5: Real-world robot observation (left) and third-person view (right), ordered as numbered.