CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization

TL;DR

A method of global localization on a map with semantic object landmarks by augmenting the correspondence matching using Vision Language Models (VLMs) and incorporating pose calculation using the weighted least squares considering correspondence similarity and observation completeness to improve the robustness.

Abstract

This letter proposes a method of global localization on a map with semantic object landmarks. One of the most promising approaches for localization on object maps is to use semantic graph matching using landmark descriptors calculated from the distribution of surrounding objects. These descriptors are vulnerable to misclassification and partial observations. Moreover, many existing methods rely on inlier extraction using RANSAC, which is stochastic and sensitive to a high outlier rate. To address the former issue, we augment the correspondence matching using Vision Language Models (VLMs). Landmark discriminability is improved by VLM embeddings, which are independent of surrounding objects. In addition, inliers are estimated deterministically using a graph-theoretic approach. We also incorporate pose calculation using the weighted least squares considering correspondence similarity and observation completeness to improve the robustness. We confirmed improvements in matching and pose estimation accuracy through experiments on ScanNet and TUM datasets.

Figures (7)

• Figure 1: We propose CLIP-Clique, an object-based RGB-D global localization method driven by a novel correspondence matching strategy. It leverages two types of graphs: (i) 3D semantic graph for accurately estimating object correspondences, and (ii) spatial compatibility graph for efficiently extracting inlier correspondences as spatially compatible sets. We augment an existing semantic graph-based method Guo2021a with a Vision Language Model, i.e., CLIP Radford2021 to enhance landmark discriminability and robustness. We also exploit CLIP-based similarity estimation in ranking multiple inlier candidates calculated as maximal cliques of the compatibility graph, and similarity-weighted least squares for accurate pose calculation.
• Figure 2: Overview of the proposed method. (i) For the given query and map landmarks, semantic graphs are built and the descriptors for each node is calculated. Specifically, we use Semantic Histograms Guo2021a and CLIP embeddings as node descriptors (Sec. \ref{['sec:proposed_method_object_descriptor']}). (ii) Correspondence candidates are generated based on the similarity of corresponding object descriptors (Sec. \ref{['sec:proposed_method_correspondence_generation']}). (iii) From the initial correspondence set, inlier sets are extracted as sets of spatially compatible correspondences using the compatibility graph and maximal clique finding. Multiple sets are scored by the sum of the similarity to evaluate the likelihood (Sec. \ref{['sec:proposed_method_inlier_extraction']}). (iv) A camera pose is calculated using the extracted inlier set. To mitigate the problem of wrong correspondences and incomplete observations, we employ weighted least squares based on the correspondence similarity and the observation completeness (Sec. \ref{['sec:proposed_method_pose_estimation']}).
• Figure 3: An illustration of the Semantic Histogram Guo2021a. All possible paths with a fixed step length (here set to 3) starting from the target node are searched and the patterns of label sequences are recorded in a histogram. It effectively encodes the topological information around the object.
• Figure 4: How to select correspondences for an observation. The horizontal axis of each subfigure corresponds the landmarks. When the similarity values are sorted, we set a threshold at the point where there is the largest gap to extract the arbitrary number of likely correspondence candidates.
• Figure 5: Relationship between the number of landmarks and the latency