UniPR-3D: Towards Universal Visual Place Recognition with Visual Geometry Grounded Transformer

TL;DR

This work tackles Visual Place Recognition (VPR) under diverse conditions by moving beyond single-view cues to multi-view geometry. It introduces UniPR-3D, a 3D-token based VPR framework built on the Visual Geometry Grounded Transformer (VGGT) that jointly leverages 2D and 3D tokens with dedicated aggregation strategies. The system supports both single-frame and variable-length sequence matching, employing GeM pooling for token-level descriptors and an optimal transport (Sinkhorn) approach for patch-level correspondences, enabling robust cross-view localization. Empirical results on benchmarks such as MSLS, Nordland, and Oxford show state-of-the-art performance, demonstrating the effectiveness and generalization of geometry-grounded tokens for VPR with practical implications for robotics and navigation.

Abstract

Visual Place Recognition (VPR) has been traditionally formulated as a single-image retrieval task. Using multiple views offers clear advantages, yet this setting remains relatively underexplored and existing methods often struggle to generalize across diverse environments. In this work we introduce UniPR-3D, the first VPR architecture that effectively integrates information from multiple views. UniPR-3D builds on a VGGT backbone capable of encoding multi-view 3D representations, which we adapt by designing feature aggregators and fine-tune for the place recognition task. To construct our descriptor, we jointly leverage the 3D tokens and intermediate 2D tokens produced by VGGT. Based on their distinct characteristics, we design dedicated aggregation modules for 2D and 3D features, allowing our descriptor to capture fine-grained texture cues while also reasoning across viewpoints. To further enhance generalization, we incorporate both single- and multi-frame aggregation schemes, along with a variable-length sequence retrieval strategy. Our experiments show that UniPR-3D sets a new state of the art, outperforming both single- and multi-view baselines and highlighting the effectiveness of geometry-grounded tokens for VPR. Our code and models will be made publicly available on Github https://github.com/dtc111111/UniPR-3D.

Figures (11)

• Figure 1: Illustration of a typical single-view VPR model (top) and our multi-view model (bottom). Single-view VPR extracts image features with a deep backbone and aggregates them into an image descriptor. In contrast, our multi-view VPR model employs a VGGT backbone to jointly extract both 2D and 3D tokens from multiple views, followed by tailored aggregation strategies for each token type. Our framework supports both single-frame and variable-length sequence matching, and achieves state-of-the-art performance across standard VPR benchmarks.
• Figure 2: Overview. We propose the first VPR method that supports both single-frame and sequence-level place retrieval. Specifically, we use DINOv2 as our visual feature extractor and then utilize the alternating attention blocks of VGGT to derive 3D tokens. The resulting 3D tokens are divided into different groups, each processed with a dedicated aggregation strategy to form the final descriptor. In particular, cls tokens and register tokens are aggregated using GeM pooling, while patch tokens are processed through an optimal transport module, where the Sinkhorn algorithm is applied to compute the assignment matrix. The outputs from these modules are finally concatenated to produce the global descriptor.
• Figure 3: Qualitative sequence matching results on the Oxford dataset oxfordrobo. The left column shows two query sequence images, while the three right columns present the top-3 candidates retrieved by our UniPR-3D and the baseline CaseVPR casevpr. Successful retrievals are framed in green, while erroneous ones are shown in red. UniPR-3D retrieves the correct place even under challenging seasonal, weather, viewpoint, and day-night variations.
• Figure 4: Activation heatmaps of 3D and 2D features in the Oxford dataset oxfordrobo. The left column shows reference images, and the two right columns their corresponding 3D and 2D feature heatmaps. Note how the different feature types fire at different locations in the image, illustrating their complementary nature.
• Figure 5: Qualitative sequence matching results on the MSLS dataset msls. The left column shows two query sequence images, while the three right columns present the top-3 candidates retrieved by our UniPR-3D and the baseline CaseVPR casevpr. Successful retrievals are framed in green, while erroneous ones are shown in red. UniPR-3D retrieves the correct place even under challenging seasonal, weather, viewpoint, and day-night variations.