CLIP-Loc: Multi-modal Landmark Association for Global Localization in Object-based Maps

TL;DR

This work tackles scalable global localization using object-based maps by labeling landmarks with natural language and leveraging a Vision-Language Model to generate cross-modal correspondences. It introduces CLIP-Loc, a three-stage pipeline that (i) generates CLIP-based correspondence candidates, (ii) performs Balanced-PROSAC inlier extraction, and (iii) estimates and optionally refines the camera pose. The key contributions are the CLIP-based candidate generation, the Balanced-PROSAC sampling strategy, and the empirical demonstration on real datasets showing improved accuracy and faster convergence over category-based baselines. The approach enhances robustness to viewpoint and appearance variations while reducing computational burden, enabling more practical localization within object-centric SLAM workflows.

Abstract

This paper describes a multi-modal data association method for global localization using object-based maps and camera images. In global localization, or relocalization, using object-based maps, existing methods typically resort to matching all possible combinations of detected objects and landmarks with the same object category, followed by inlier extraction using RANSAC or brute-force search. This approach becomes infeasible as the number of landmarks increases due to the exponential growth of correspondence candidates. In this paper, we propose labeling landmarks with natural language descriptions and extracting correspondences based on conceptual similarity with image observations using a Vision Language Model (VLM). By leveraging detailed text information, our approach efficiently extracts correspondences compared to methods using only object categories. Through experiments, we demonstrate that the proposed method enables more accurate global localization with fewer iterations compared to baseline methods, exhibiting its efficiency.

Figures (7)

• Figure 1: In conventional global localization in object-based maps, only class information is utilized for constructing a correspondence candidate set, leading to exponential growth of the number as the landmark increases. In this paper, we propose to assign a natural language description as a label to each landmark, and match them with visual observation using a Vision Language Model (VLM). It enables more efficient correspondence matching leveraging the fine-grained information given by the text labels.
• Figure 2: Overview of the proposed method. We assume that object landmarks in the map are labeled by natural language descriptions. In addition, for a query image, object locations and classes are given by a general object detector (here we use YOLOv8). Given those data, a camera pose is estimated in the following steps: 1) The query objects and the landmark labels are embedded to a common feature space using image and text encoders of CLIP Radford2021, respectively. For each image embedding, $k$ nearest neighboring text embeddings are retrieved, and correspondence candidates are generated. 2) Inlier correspondences are estimated through B-PROSAC described in Sec. \ref{['sec:proposed_method_inlier_extraction']}. It consists of sampling three candidates considering the matching score, pose estimation via P4P, and pose verification. The final pose is given as the one with the best score after a certain number of iterations. 3) Optionally, the pose can be further refined by methods, e.g., Zins2022a.
• Figure 3: Sorting in the proposed method. Top 1 nearest landmarks for the observations are first sorted by the score and stored in an ordered correspondence candidate list. Top 2 landmarks are then sorted and concatenated to the list, and so forth.
• Figure 4: Success rate with regard to error thresholds when the number of iteration is 500. The proposed methods (hybrid_prosac/hybrid_b-prosac) outperformed the category-based baseline.
• Figure 5: Success rate with different numbers of iterations. hybrid_prosac and hybrid_b-prosac consistently outperformed the counterparts relying on category information and RANSAC.