Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective

TL;DR

The paper addresses attention collapse in VGGT's global self-attention when processing long sequences. It introduces a mean-field PDE framework that models token-feature diffusion on the sphere, showing a Dirac-type steady state is approached at rate $O(1/L)$ and that token merging slows diffusion by effectively reducing token count, delaying collapse while reducing computation. The main contributions are a quantitative diffusion-based explanation linking entropy decay, rank contraction, and token-merging, closed-form predictions that match ScanNet-50 experiments, and practical guidance for scalable 3D vision transformers. This framework provides a principled design lens for future cross-view, multi-modal transformers and clarifies how efficiency tricks can be integrated without retraining to preserve geometric fidelity.

Abstract

Visual Geometry Grounded Transformer (VGGT) delivers state-of-the-art feed-forward 3D reconstruction, yet its global self-attention layer suffers from a drastic collapse phenomenon when the input sequence exceeds a few hundred frames: attention matrices rapidly become near rank-one, token geometry degenerates to an almost one-dimensional subspace, and reconstruction error accumulates super-linearly.In this report,we establish a rigorous mathematical explanation of the collapse by viewing the global-attention iteration as a degenerate diffusion process.We prove that,in VGGT, the token-feature flow converges toward a Dirac-type measure at a $O(1/L)$ rate, where $L$ is the layer index, yielding a closed-form mean-field partial differential equation that precisely predicts the empirically observed rank profile.The theory quantitatively matches the attention-heat-map evolution and a series of experiments outcomes reported in relevant works and explains why its token-merging remedy -- which periodically removes redundant tokens -- slows the effective diffusion coefficient and thereby delays collapse without additional training.We believe the analysis provides a principled lens for interpreting future scalable 3D-vision transformers,and we highlight its potential for multi-modal generalization.

Figures (4)

• Figure 1: Architectural Overview of the evolution of VGGT to FastVGGT and Attention Collapse Phenomenon.The model iteratively refines features through global attention with token merging and local self-attention with token unmerging. The output visualizes attention heatmaps across different transformer blocks for selected tokens, ultimately demonstrating the attention collapse that occurs during global attention procedure.
• Figure 2: Quantitative evaluation on the ScanNet-50 dataset. We report camera and geometry metrics, including Absolute Rotation Error (ARE), Aligned Relative Pose Error in Translation (RPE-Trans), Rotation (RPE-Rot), and Absolute Trajectory Error (ATE), across different token merge ratios. All metrics consistently improve as the merge ratio increases, indicating that diffusion-regularized attention not only delays collapse but also enhances geometric stability and reconstruction accuracy of VGGT. These results demonstrate that the proposed theoretical regularization translates directly into empirical performance gains on real-world 3D scenes.
• Figure 3: Comparison between theoretical prediction and experimental measurement of entropy and rank. Left: normalized entropy; Right: effective rank versus fusion strength $m$. Blue: empirical results on ScanNet-50; Red: theoretical predictions from the mean-field diffusion model. Both curves align closely, showing that token merging slows attention collapse and preserves feature diversity as predicted.
• Figure 4: Visualization of attention collapse across transformer blocks. Shown are attention heatmaps for selected tokens at different depths.As we merged tokens,the rank collapse in global attention has been alleviated