HTTM: Head-wise Temporal Token Merging for Faster VGGT

TL;DR

VGGT enables direct joint inference of camera poses, depths, and geometry but struggles with latency due to global attention over long token sequences. HTTM introduces training-free, head-wise temporal token merging with temporal reordering and adaptive outlier filtering to accelerate VGGT's global attention while preserving accuracy, achieving up to 7× speedups on long sequences. By exploiting head-level similarity patterns and spatio-temporal redundancy, HTTM reduces merging cost and maintains token diversity, outperforming or matching previous fast VGGT variants on challenging datasets. This approach enables scalable, efficient 3D reconstruction from many views, making VGGT practical for larger scenes and longer sequences in real-time or near-real-time contexts.

Abstract

The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass. However, this joint inference mechanism requires global attention layers that perform all-to-all attention computation on tokens from all views. For reconstruction of large scenes with long-sequence inputs, this causes a significant latency bottleneck. In this paper, we propose head-wise temporal merging (HTTM), a training-free 3D token merging method for accelerating VGGT. Existing merging techniques merge tokens uniformly across different attention heads, resulting in identical tokens in the layers' output, which hinders the model's representational ability. HTTM tackles this problem by merging tokens in multi-head granularity, which preserves the uniqueness of feature tokens after head concatenation. Additionally, this enables HTTM to leverage the spatial locality and temporal correspondence observed at the head level to achieve higher merging ratios with lower merging costs compared to existing methods. Thus, HTTM achieves up to 7x acceleration with negligible performance drops in a GPU-based inference.

Figures (13)

• Figure 1: HTTM forms spatio-temporal merging blocks that jointly consider neighboring tokens across consecutive frames. This design exploits temporal coherence and spatial redundancy to merge tokens efficiently. With the same merging ratio, HTTM reduces the merging cost by 4.58$\times$
• Figure 2: Attention score distribution comparison between VGGT and Llama 3.1 8Bgrattafiori2024llama3herdmodels. The distribution of attention scores in VGGT is heavily concentrated around low values in both its early and late layers. In the middle layers, attention distribution is still more skewed towards lower values compared to Llama.
• Figure 3: Cosine similarity patterns averaged across all heads between query tokens of 4 adjacent frames. High similarities observed along the block diagonals indicate that tokens within the same spatial region (local areas) and corresponding locations across consecutive frames share highly similar features.
• Figure 4: Cosine similarity between query tokens at the 14th deep global attention layer in single-frame reconstruction. The high visual redundancy frame (a wall) shows stronger spatial similarity compared to the low visual redundancy frame (cluttered objects).
• Figure 5: Cosine similarity between query tokens of 8 adjacent frames at the 14th global attention layer. High similarity between input frames leads to high temporal similarity of query tokens, as shown by the high scores on off-diagonals.