CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Framework

TL;DR

CMMLoc tackles the problem of locating a 3D urban position described by natural language by explicitly modeling partial relevance between text and 3D objects. It introduces a Cauchy-Mixture-Model based Transformer (CMMT) with a spatial consolidation scheme for robust coarse submap retrieval, and a fine localization stage that employs a pre-alignment strategy plus Cardinal Direction Integration to refine cross-modal alignment. On KITTI360Pose, CMMLoc achieves state-of-the-art performance in both coarse retrieval and fine localization, and demonstrates robustness to semantic-label noise through comprehensive ablations. The approach advances practical text-to-point-cloud localization for outdoor scenes, with potential applications in autonomous navigation and urban robotics.

Abstract

The goal of point cloud localization based on linguistic description is to identify a 3D position using textual description in large urban environments, which has potential applications in various fields, such as determining the location for vehicle pickup or goods delivery. Ideally, for a textual description and its corresponding 3D location, the objects around the 3D location should be fully described in the text description. However, in practical scenarios, e.g., vehicle pickup, passengers usually describe only the part of the most significant and nearby surroundings instead of the entire environment. In response to this $\textbf{partially relevant}$ challenge, we propose $\textbf{CMMLoc}$, an uncertainty-aware $\textbf{C}$auchy-$\textbf{M}$ixture-$\textbf{M}$odel ($\textbf{CMM}$) based framework for text-to-point-cloud $\textbf{Loc}$alization. To model the uncertain semantic relations between text and point cloud, we integrate CMM constraints as a prior during the interaction between the two modalities. We further design a spatial consolidation scheme to enable adaptive aggregation of different 3D objects with varying receptive fields. To achieve precise localization, we propose a cardinal direction integration module alongside a modality pre-alignment strategy, helping capture the spatial relationships among objects and bringing the 3D objects closer to the text modality. Comprehensive experiments validate that CMMLoc outperforms existing methods, achieving state-of-the-art results on the KITTI360Pose dataset. Codes are available in this GitHub repository https://github.com/kevin301342/CMMLoc.

Figures (10)

• Figure 1: Given a text description of a location in (a), CMMLoc searches the 3D city and identifies the most likely target location of the described position within a submap in (b). Notably, text descriptions do not correspond to all objects within the submap, where the irrelevant objects are shown in gray color.
• Figure 2: The overview of proposed CMMLoc. It is a coarse-to-fine architecture consisting of two stages: Coarse submap retrieval and Fine localization. Coarse submap retrieval. Given text descriptions, we first identify a set of candidate submaps potentially containing the target position. This is achieved by retrieving the Top-k nearest submaps from a constructed database of submaps using our CMM-based retrieval model. Fine localization. We then refine the coordinates of the retrieved submaps via our pre-alignment strategy and cardinal direction integration module to improve localization accuracy.
• Figure 3: Illustration of coarse submap retrieval. We introduce the CMM Transformer and spatial consolidation scheme in the object encoding branch to model the partial relevance between 3D objects and achieve a better representation of the submap. Note that the T5 model in the text encoding branch is frozen during this process.
• Figure 4: Illustration of CMM Transformer.
• Figure 5: Illustration of Cardinal Direction Integration.