MSSPlace: Multi-Sensor Place Recognition with Visual and Text Semantics

TL;DR

This work tackles robust place recognition for autonomous navigation by proposing MSSPlace, a modular late-fusion architecture that jointly encodes images from multiple cameras, LiDAR point clouds, semantic masks, and textual descriptions into a single global descriptor. The authors extend Oxford RobotCar and NCLT with semantic masks (via OneFormer) and natural language scene descriptions (via MiniGPT-4) and perform extensive ablations to quantify each modality's contribution. Results show that multi-sensor fusion substantially improves recognition accuracy and, in some configurations, achieves state-of-the-art performance on both datasets, while individual modalities (semantic masks or text) can be competitive on their own. The work provides practical insights for designing scalable, reproducible, multimodal place recognition systems and releases code for reproducibility and further research.

Abstract

Place recognition is a challenging task in computer vision, crucial for enabling autonomous vehicles and robots to navigate previously visited environments. While significant progress has been made in learnable multimodal methods that combine onboard camera images and LiDAR point clouds, the full potential of these methods remains largely unexplored in localization applications. In this paper, we study the impact of leveraging a multi-camera setup and integrating diverse data sources for multimodal place recognition, incorporating explicit visual semantics and text descriptions. Our proposed method named MSSPlace utilizes images from multiple cameras, LiDAR point clouds, semantic segmentation masks, and text annotations to generate comprehensive place descriptors. We employ a late fusion approach to integrate these modalities, providing a unified representation. Through extensive experiments on the Oxford RobotCar and NCLT datasets, we systematically analyze the impact of each data source on the overall quality of place descriptors. Our experiments demonstrate that combining data from multiple sensors significantly improves place recognition model performance compared to single modality approaches and leads to state-of-the-art quality. We also show that separate usage of visual or textual semantics (which are more compact representations of sensory data) can achieve promising results in place recognition. The code for our method is publicly available: https://github.com/alexmelekhin/MSSPlace

Figures (7)

• Figure 1: An overview of a multi-sensor place recognition pipeline with visual and text semantics. First, the input data is encoded into a query descriptor. A K-nearest neighbors search is then performed between the query and the database. Finally, the position of the closest database descriptor found is considered as the answer.
• Figure 2: High-level overview of the proposed multimodal MSSPlace method. The method has a modular architecture and consists of four branches: the Image Encoder, Semantic Masks Encoder, Text Encoder, and Point Cloud Encoder. Each branch encodes the input data into a descriptor, capturing the essential information specific to its respective modality. Subsequently, a descriptor aggregation step is performed to combine these individual descriptors and obtain the global place descriptor, which represents the comprehensive characteristics of the vehicle place.
• Figure 3: Image and semantic encoder scheme. Both encoders have the same architecture based on ResNet18 with difference in input channels number: (a) ResNet18-based encoder architecture; (b) Different setups of 1st ResNet convolutional layer: (I) - for RGB images, (II) - for semantic masks.
• Figure 4: Text encoders: (a) TF-IDF-based; (b) CLIP text encoder-based.
• Figure 5: Embedding Fusion modules