ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition

TL;DR

This work introduces a fast and lightweight framework to encode images and point clouds into place-distinctive descriptors and proposes an effective Field of View (FoV) transformation module to convert point clouds into an analogous modality as images.

Abstract

Place recognition is an important task for robots and autonomous cars to localize themselves and close loops in pre-built maps. While single-modal sensor-based methods have shown satisfactory performance, cross-modal place recognition that retrieving images from a point-cloud database remains a challenging problem. Current cross-modal methods transform images into 3D points using depth estimation for modality conversion, which are usually computationally intensive and need expensive labeled data for depth supervision. In this work, we introduce a fast and lightweight framework to encode images and point clouds into place-distinctive descriptors. We propose an effective Field of View (FoV) transformation module to convert point clouds into an analogous modality as images. This module eliminates the necessity for depth estimation and helps subsequent modules achieve real-time performance. We further design a non-negative factorization-based encoder to extract mutually consistent semantic features between point clouds and images. This encoder yields more distinctive global descriptors for retrieval. Experimental results on the KITTI dataset show that our proposed methods achieve state-of-the-art performance while running in real time. Additional evaluation on the HAOMO dataset covering a 17 km trajectory further shows the practical generalization capabilities. We have released the implementation of our methods as open source at: https://github.com/haomo-ai/ModaLink.git.

Figures (11)

• Figure 1: The purpose of ModaLink is to retrieve the most positive corresponding point cloud of the query image in a pre-built large-scale point cloud database.
• Figure 2: The training framework of our proposed ModaLink framework. Point clouds are converted to depth images via projection. Then, the query image and depth images are cropped into the same overlap of FoV. Based on the depth completion module, sparse depth images are upsampled to dense depth images. Then, global place descriptors are generated by a shared-weight encoder. Finally, we adopt triplet loss for supervision.
• Figure 3: Visualization of image and point cloud alignment based on intrinsic and extrinsic matrix. After alignment, we crop the images based on FoV overlap and complete sparse depth images to generate dense depth images.
• Figure 4: Three cases of depth image completion. The red point represents a pixel on the image plane where no LiDAR point is projected.
• Figure 5: A demonstration of depth image interpolation.