GPA-VGGT:Adapting VGGT to Large Scale Localization by Self-Supervised Learning with Geometry and Physics Aware Loss

TL;DR

This work tackles large-scale camera pose and scene geometry estimation under unlabeled data. It extends the Visual Geometry Grounded Transformer (VGGT) with multi-sequence 3D self-supervision by employing temporal sliding windows and physics-aware losses that enforce photometric and geometric consistency across multiple frames, along with robust view selection and auto-masking. The proposed GPA-VGGT achieves rapid convergence and superior pose accuracy and depth quality on KITTI sequences, outperforming both monocular self-supervised and several supervised geometry baselines under the same windowing protocol. The approach demonstrates that carefully designed loss constraints can transfer strong geometric reasoning to transformer-based architectures without architectural changes, enabling scalable, robust large-scale localization. This loss-driven framework has practical implications for outdoor SLAM, autonomous driving, and 3D scene understanding in unlabeled, dynamic environments.

Abstract

Transformer-based general visual geometry frameworks have shown promising performance in camera pose estimation and 3D scene understanding. Recent advancements in Visual Geometry Grounded Transformer (VGGT) models have shown great promise in camera pose estimation and 3D reconstruction. However, these models typically rely on ground truth labels for training, posing challenges when adapting to unlabeled and unseen scenes. In this paper, we propose a self-supervised framework to train VGGT with unlabeled data, thereby enhancing its localization capability in large-scale environments. To achieve this, we extend conventional pair-wise relations to sequence-wise geometric constraints for self-supervised learning. Specifically, in each sequence, we sample multiple source frames and geometrically project them onto different target frames, which improves temporal feature consistency. We formulate physical photometric consistency and geometric constraints as a joint optimization loss to circumvent the requirement for hard labels. By training the model with this proposed method, not only the local and global cross-view attention layers but also the camera and depth heads can effectively capture the underlying multi-view geometry. Experiments demonstrate that the model converges within hundreds of iterations and achieves significant improvements in large-scale localization. Our code will be released at https://github.com/X-yangfan/GPA-VGGT.

Figures (5)

• Figure 1: Comparison of traditional self-supervised learning, VGGT, and our multi-sequence 3D self-supervised framework, which induces large-scale geometric reasoning in VGGT through structured loss design without architectural changes.
• Figure 2: (a) VGG-T–based network architecture with a depth head and a camera head for predicting depth and relative camera poses. (b) Multi-frame projection from source frames to a keyframe with validity mask construction. (c) Per-pixel hard source selection, where the source pixel with the minimum photometric–geometric cost is chosen for supervision.
• Figure 3: Chained inference with overlapping windows, where shared frames enable forward propagation of pose predictions to form long-range trajectories.
• Figure 4: Trajectory comparison results: (a) trajectory comparison on Sequence 07 of the KITTI dataset; (b) trajectory comparison on Sequence 09 of the KITTI dataset.
• Figure 5: Comparison of depth predictions from different models.