XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

TL;DR

XStreamVGGT tackles the memory bottleneck of streaming 3D vision transformers by pruning redundant multi-view KV cache tokens and applying per-channel Key and per-token Value quantization to a fixed budget. The method preserves geometric references from the first frame while updating evidence with the current frame, yielding substantial memory reduction ($4.42\times$) and speedups ($5.48\times$) with mostly negligible performance loss across video depth estimation, 3D reconstruction, and camera pose tasks. By tightly integrating pruning with distribution-aware quantization, XStreamVGGT enables scalable, tuning-free streaming inference for 3D vision applications. This approach offers practical benefits for real-world streaming scenarios and sets the stage for adaptive cache budgeting in future work.

Abstract

Learning-based 3D visual geometry models have benefited substantially from large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention for strong streaming reconstruction, but suffers from unbounded KV cache growth, leading to escalating memory consumption and inference latency as input frames accumulate. We propose XStreamVGGT, a tuning-free approach that systematically compresses the KV cache through joint pruning and quantization, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs originating from multi-view inputs are pruned through efficient token importance identification, enabling a fixed memory budget. Leveraging the unique distribution of KV tensors, we incorporate KV quantization to further reduce memory consumption. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling scalable and practical streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

Figures (6)

• Figure 1: Efficiency analysis on a single 80GB A100 GPU. As the number of input frames increases, StreamVGGT and VGGT exhibit rapid FPS degradation and quickly encounter out-of-memory (OOM) errors, whereas XStreamVGGT consistently maintains higher FPS without OOM.
• Figure 2: Overview of XStreamVGGT. Upon receiving a new input frame (Step 1), Queries from the global attention layer are aggregated via average pooling to form a compact representation, which is then matched against the Key to estimate token importance (Step 2). Guided by these Key-derived importance scores, low-importance historical KV pairs are selectively pruned, while KVs from the first frame are explicitly retained to preserve geometric consistency. The remaining high-importance KVs are concatenated with the first-frame KVs and the newly generated KVs from the current frame (Step 3). Finally, the updated KV cache is compressed via quantization yielding a compact cache for subsequent updates (Steps 4–5).
• Figure 3: Magnitude distributions of the Key and Value. The Key exhibits pronounced channel-wise outliers, with a small number of channels having larger magnitudes than the rest.
• Figure 4: Ablation study of cache length.
• Figure 5: Analysis of memory with increasing frame length.