Online Embedding Multi-Scale CLIP Features into 3D Maps

TL;DR

The paper tackles open-vocabulary semantic mapping for 3D maps to support language-based queries in unknown environments. It introduces CLIPMapper, an online system that embeds multi-scale CLIP features into 3D maps by patching RGB images at multiple scales and batch-processing patches for a single forward pass, then back-projects depth to world coordinates to populate a semantically rich map. The approach enables offline retrieval via text queries and supports a zero-shot object-goal navigation pipeline that uses CLIP-based localization to guide exploration with a multi-goal planner. Across simulation (Habitat) and real-robot experiments (Vizbot), the method achieves faster mapping and higher success rates, including for objects outside standard COCO vocabularies, and demonstrates robust open-vocabulary object retrieval and multi-object navigation, illustrating practical impact for autonomous navigation and semantic mapping.

Abstract

This study introduces a novel approach to online embedding of multi-scale CLIP (Contrastive Language-Image Pre-Training) features into 3D maps. By harnessing CLIP, this methodology surpasses the constraints of conventional vocabulary-limited methods and enables the incorporation of semantic information into the resultant maps. While recent approaches have explored the embedding of multi-modal features in maps, they often impose significant computational costs, lacking practicality for exploring unfamiliar environments in real time. Our approach tackles these challenges by efficiently computing and embedding multi-scale CLIP features, thereby facilitating the exploration of unfamiliar environments through real-time map generation. Moreover, the embedding CLIP features into the resultant maps makes offline retrieval via linguistic queries feasible. In essence, our approach simultaneously achieves real-time object search and mapping of unfamiliar environments. Additionally, we propose a zero-shot object-goal navigation system based on our mapping approach, and we validate its efficacy through object-goal navigation, offline object retrieval, and multi-object-goal navigation in both simulated environments and real robot experiments. The findings demonstrate that our method not only exhibits swifter performance than state-of-the-art mapping methods but also surpasses them in terms of the success rate of object-goal navigation tasks.

Figures (7)

• Figure 1: Online Embedding Multi-Scale CLIP Features into 3D Maps. Our approach facilitates real-time object exploration in unknown environments through efficient computation. Additionally, embedding CLIP features into the resulting map enables offline retrieval from the map post-creation, thus augmenting the practical utility of the proposed method.
• Figure 2: Embedding multi-scale CLIP features into a 3D map.
• Figure 3: System implementation of object-goal navigation based on the proposed CLIP feature embedding method.
• Figure 4: Qualitative results of object-goal navigation using our mapping method (ViT-L/14). From left to right: RGB image, depth image, and 2D obstacle map. The red circle on the map indicates the robot's position, the red line represents the robot's trajectory, the green circle denotes the detected object's position, and the green line marks the path to the object's position. This demonstrates our method's ability to locate objects specified by arbitrary text.
• Figure 5: Comparison results of object retrieval. The results suggest that our method exhibits better retrieval ability for spatial queries than VLMap.