IVLMap: Instance-Aware Visual Language Grounding for Consumer Robot Navigation

TL;DR

The paper tackles the challenge of instance- and attribute-level VLN by introducing IVLMap, a bird’s-eye semantic map that fuses RGB-D data with a natural-language indexing scheme and SAM-based instance segmentation. By augmenting VLMap with per-instance masks and color attributes, and combining open-vocabulary landmark localization with LLM-guided parsing, IVLMap enables precise localization and zero-shot navigation to specific object instances described in natural language. The approach is validated through extensive experiments in Habitat and Matterport3D settings, showing superior navigation accuracy and robust zero-shot performance compared with baselines like VLMap, CoW, and CLIP Map. This work advances practical robot navigation by enabling fine-grained, language-guided instance grounding in real-world environments, with future work targeting dynamic scenes and 3D semantic maps.

Abstract

Vision-and-Language Navigation (VLN) is a challenging task that requires a robot to navigate in photo-realistic environments with human natural language promptings. Recent studies aim to handle this task by constructing the semantic spatial map representation of the environment, and then leveraging the strong ability of reasoning in large language models for generalizing code for guiding the robot navigation. However, these methods face limitations in instance-level and attribute-level navigation tasks as they cannot distinguish different instances of the same object. To address this challenge, we propose a new method, namely, Instance-aware Visual Language Map (IVLMap), to empower the robot with instance-level and attribute-level semantic mapping, where it is autonomously constructed by fusing the RGBD video data collected from the robot agent with special-designed natural language map indexing in the bird's-in-eye view. Such indexing is instance-level and attribute-level. In particular, when integrated with a large language model, IVLMap demonstrates the capability to i) transform natural language into navigation targets with instance and attribute information, enabling precise localization, and ii) accomplish zero-shot end-to-end navigation tasks based on natural language commands. Extensive navigation experiments are conducted. Simulation results illustrate that our method can achieve an average improvement of 14.4\% in navigation accuracy. Code and demo are released at https://ivlmap.github.io/.

Figures (12)

• Figure 1: Under the guidance of our IVLMap, robotic agents can accomplish instance-level target navigation. Leveraging the map's information, the robot can navigate to specific objects based on their instance attributes(color, shape, etc). This capability enables the robotic agent to execute navigation tasks with a higher degree of precision, enhancing its ability to reach designated objects accurately.
• Figure 2: The IVLMap pipeline consists of two main components. The first focuses on 3D reconstruction and constructing a visual language map. Building on this foundation, the second part integrates the Segment Anything Model (SAM), enhancing the map's representation with a segmentation-aware approach for more detailed information.
• Figure 3: A schematic diagram illustrating a Matching Algorithm. We matched the masks generated by the SAM model with the Pixel-Text Similarity obtained from VLMap, assigning labels to each mask. This facilitated the subsequent implementation of IVLMap.
• Figure 4: We created an Interactive Dataset Collection Scheme by combining the cmu-exploration development environment with the Habitat simulator. This involves integrating cmu-exploration's autonomous exploration with Habitat robot agents for a unified dataset collection approach.
• Figure 5: 3D Reconstruction Map in Bird's-Eye View