Reloc-VGGT: Visual Re-localization with Geometry Grounded Transformer

TL;DR

Reloc-VGGT tackles visual re-localization by injecting multi-view spatial cues through an early fusion with a 3D foundation model and a pose tokenizer. It introduces learnable Fourier pose embeddings and a camera-token projection to align 3D pose tokens with 2D patch tokens, paired with a sparse mask attention mechanism that reduces $O(n^2)$ global attention to $O(5n-5)$ for scalable inference. Trained on approximately eight million posed image pairs, the approach achieves state-of-the-art or competitive results across indoor and outdoor relocalization benchmarks, including strong generalization to unseen environments. The work demonstrates that combining multi-view geometry with efficient transformer-based fusion enables robust, real-time visual localization at scale.

Abstract

Visual localization has traditionally been formulated as a pair-wise pose regression problem. Existing approaches mainly estimate relative poses between two images and employ a late-fusion strategy to obtain absolute pose estimates. However, the late motion average is often insufficient for effectively integrating spatial information, and its accuracy degrades in complex environments. In this paper, we present the first visual localization framework that performs multi-view spatial integration through an early-fusion mechanism, enabling robust operation in both structured and unstructured environments. Our framework is built upon the VGGT backbone, which encodes multi-view 3D geometry, and we introduce a pose tokenizer and projection module to more effectively exploit spatial relationships from multiple database views. Furthermore, we propose a novel sparse mask attention strategy that reduces computational cost by avoiding the quadratic complexity of global attention, thereby enabling real-time performance at scale. Trained on approximately eight million posed image pairs, Reloc-VGGT demonstrates strong accuracy and remarkable generalization ability. Extensive experiments across diverse public datasets consistently validate the effectiveness and efficiency of our approach, delivering high-quality camera pose estimates in real time while maintaining robustness to unseen environments. Our code and models will be publicly released upon acceptance.https://github.com/dtc111111/Reloc-VGGT.

Figures (5)

• Figure 1: Illustration of an existing visual relocalization baseline (top) and our proposed method (bottom). The upper pipeline illustrates conventional relocalization methods, which rely on pair-wise pose regression and employ late fusion to estimate camera poses. In contrast, our approach introduces a multi-frame relocalization framework that performs early fusion through spatial information injection, enabling more effective spatial feature integration. Our method achieves state-of-the-art performance across standard visual relocalization and pose regression benchmarks.
• Figure 2: System Overview. We propose a novel visual re-localization framework with 3D foundation model. We introduce a novel pose tokenizer and projection module to better leverage the spatial information for early fusion and design a sparse mask attention strategy to enhance test-time inference speed and reduce computational complexity from quadratic to linear.
• Figure 3: We visualize the global attention maps under different masking strategies. From left to right, the figures correspond to the original global attention in VGGT, the causal attention in StreamVGGT streamvggt, and our proposed sparse mask attention and dilated mask attention.
• Figure 4: We visualize pose estimates for two scenes: stairs from the 7 Scenes dataset 7scenes and GreatCourt from Cambridge Landmarks cambridge. We compare our results with the exisitng sota method Reloc3R reloc3r.
• Figure 5: We visualize the runtime trend as the number of selected top-$k$ images increases, comparing the global attention mechanism with our proposed sparse mask attention. Our sparse mask attention strategy significantly reduces inference time, making relocalization over longer sequences computationally feasible.