PlaceFormer: Transformer-based Visual Place Recognition using Multi-Scale Patch Selection and Fusion

TL;DR

PlaceFormer addresses robust visual place recognition under long-term appearance changes by using a Vision Transformer backbone to obtain global descriptors and by fusing patch tokens into multi-scale patches. Attention-guided key-patch selection enables efficient, scale-aware matching, followed by mutual nearest-neighbor correspondences and geometric verification to produce a robust spatial score for re-ranking. The approach demonstrates state-of-the-art or competitive performance across multiple VPR benchmarks with favorable latency and memory profiles, while highlighting areas for further speedups in patch matching. Overall, PlaceFormer offers a practical, transformer-based solution that improves retrieval accuracy and scalability for large-scale VPR deployments.

Abstract

Visual place recognition is a challenging task in the field of computer vision, and autonomous robotics and vehicles, which aims to identify a location or a place from visual inputs. Contemporary methods in visual place recognition employ convolutional neural networks and utilize every region within the image for the place recognition task. However, the presence of dynamic and distracting elements in the image may impact the effectiveness of the place recognition process. Therefore, it is meaningful to focus on task-relevant regions of the image for improved recognition. In this paper, we present PlaceFormer, a novel transformer-based approach for visual place recognition. PlaceFormer employs patch tokens from the transformer to create global image descriptors, which are then used for image retrieval. To re-rank the retrieved images, PlaceFormer merges the patch tokens from the transformer to form multi-scale patches. Utilizing the transformer's self-attention mechanism, it selects patches that correspond to task-relevant areas in an image. These selected patches undergo geometric verification, generating similarity scores across different patch sizes. Subsequently, spatial scores from each patch size are fused to produce a final similarity score. This score is then used to re-rank the images initially retrieved using global image descriptors. Extensive experiments on benchmark datasets demonstrate that PlaceFormer outperforms several state-of-the-art methods in terms of accuracy and computational efficiency, requiring less time and memory.

Figures (3)

• Figure 1: PlaceFormer leverages patches of varying scales achieved through the fusion of patch tokens in the vision transformer. From these fused patches, key patches (boxes of different colors) are selectively chosen based on the attention scores from the transformer corresponding to that patch. The model then estimates correspondences between key patches of different scales in both the query and reference images which is used for the image retrieval process.
• Figure 2: The proposed framework, PlaceFormer, encompasses a two-phase approach for visual place recognition. In the global retrieval phase, patch tokens extracted from the vision transformer undergo pooling and are subsequently processed through a linear layer, resulting in a feature vector utilized for efficient global retrieval. In the re-ranking phase, the patch tokens and attention map from the transformer's last layer are fused to generate patches at multiple scales. Leveraging attention scores, key patches are selectively identified, and correspondences between patches of different scales are computed. In the figure, for brevity, only a few inliers between the patches have been visualized. These inliers contribute to computing a spatial matching score, which is crucial for the re-ranking process.
• Figure 3: Visualization of global retrieval and re-ranking results (red box- incorrect retrieval and orange box - correct retrieval); the attention maps and key patches for query and closest retrieved image at multiple scales; and the inliers estimated across patches of various scales.