AVGGT: Rethinking Global Attention for Accelerating VGGT

Xianbing Sun; Zhikai Zhu; Zhengyu Lou; Bo Yang; Jinyang Tang; Liqing Zhang; He Wang; Jianfu Zhang

Paper

AVGGT: Rethinking Global Attention for Accelerating VGGT

Abstract

Since DUSt3R, models such as VGGT and

have shown strong multi-view 3D performance, but their heavy reliance on global self-attention results in high computational cost. Existing sparse-attention variants offer partial speedups, yet lack a systematic analysis of how global attention contributes to multi-view reasoning. In this paper, we first conduct an in-depth investigation of the global attention modules in VGGT and

to better understand their roles. Our analysis reveals a clear division of roles in the alternating global-frame architecture: early global layers do not form meaningful correspondences, middle layers perform cross-view alignment, and last layers provide only minor refinements. Guided by these findings, we propose a training-free two-step acceleration scheme: (1) converting early global layers into frame attention, and (2) subsampling global attention by subsampling K/V over patch tokens with diagonal preservation and a mean-fill component. We instantiate this strategy on VGGT and

and evaluate across standard pose and point-map benchmarks. Our method achieves up to

speedup in inference time while matching or slightly improving the accuracy of the original models, and remains robust even in extremely dense multi-view settings where prior sparse-attention baselines fail.