FastVGGT: Training-Free Acceleration of Visual Geometry Transformer

TL;DR

FastVGGT identifies Global Attention as VGGT's primary bottleneck for long image sequences and leverages a training-free token-merging strategy tailored to 3D reconstruction. By partitioning tokens into destination, salient, and source groups and applying region-based sampling, it reduces attention computations while preserving reconstruction fidelity. VRAM-efficient VGGT* enables processing of up to 1000-frame sequences with substantial speedups (up to 4x) and mitigates error accumulation in long sequences. Extensive experiments across ScanNet-50, NRGBD, and 7 Scenes demonstrate improved scalability for camera pose estimation and dense 3D reconstruction without sacrificing accuracy.

Abstract

Foundation models for 3D vision have recently demonstrated remarkable capabilities in 3D perception. However, scaling these models to long-sequence image inputs remains a significant challenge due to inference-time inefficiency. In this work, we present a detailed analysis of VGGT, a state-of-the-art feed-forward visual geometry model and identify its primary bottleneck. Visualization further reveals a token collapse phenomenon in the attention maps. Motivated by these findings, we explore the potential of token merging in the feed-forward visual geometry model. Owing to the unique architectural and task-specific properties of 3D models, directly applying existing merging techniques proves challenging. To this end, we propose FastVGGT, which, for the first time, leverages token merging in the 3D domain through a training-free mechanism for accelerating VGGT. we devise a unique token partitioning strategy tailored to 3D architectures and tasks, effectively eliminating redundant computation while preserving VGGT's powerful reconstruction capacity. Extensive experiments on multiple 3D geometry benchmarks validate the effectiveness of our approach. Notably, with 1000 input images, FastVGGT achieves a 4x speedup over VGGT while mitigating error accumulation in long-sequence scenarios. These findings underscore the potential of token merging as a principled solution for scalable 3D vision systems. Code is available at: https://mystorm16.github.io/fastvggt/.

Figures (5)

• Figure 1: We propose FastVGGT, a training-free framework that processes 1,000 images in a single inference, achieving 4$\times$ faster while mitigating error accumulation. VGGT* refers to VRAM-Efficient of VGGT, enabling larger inputs.
• Figure 2: Component-wise analysis of VGGT inference time. As the number of input frames grows, the Global Attention module increasingly dominates the computational cost.
• Figure 3: Visualizations of the Global Attention maps in VGGT, using six representative tokens (including the camera token and several image tokens), show that at every stage the attention patterns of different tokens exhibit a strong degree of similarity.
• Figure 4: Overview of the proposed token merging strategy. The pipeline begins with tokenization of input frames. To alleviate the Global Attention bottleneck, we design a five-step procedure: (1) fix initial-frame tokens as destination (Dst) tokens, serving as the global reference for spatial consistency, (2) retain the top-k salient tokens to strengthen correspondences, (3) apply region-based random sampling for balanced selection, (4) fuse source (Src) tokens into their nearest Dst tokens during attention, and (5) decouple tokens via unmerging for dense reconstruction. G-Attn and F-Attn denote Global Attention and Frame Attention, respectively.
• Figure 5: Comparison of pose estimation performance between FastVGGT and VGGT.