Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams

Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, Zhipeng Zhang

TL;DR

The Long3D benchmark is introduced, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames, and provides the definitive evaluation platform for future research in long-term 3D geometry understanding.

Abstract

The grand vision of enabling persistent, large-scale 3D visual geometry understanding is shackled by the irreconcilable demands of scalability and long-term stability. While offline models like VGGT achieve inspiring geometry capability, their batch-based nature renders them irrelevant for live systems. Streaming architectures, though the intended solution for live operation, have proven inadequate. Existing methods either fail to support truly infinite-horizon inputs or suffer from catastrophic drift over long sequences. We shatter this long-standing dilemma with InfiniteVGGT, a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache. Capitalizing on this, we devise a training-free, attention-agnostic pruning strategy that intelligently discards obsolete information, effectively ``rolling'' the memory forward with each new frame. Fully compatible with FlashAttention, InfiniteVGGT finally alleviates the compromise, enabling infinite-horizon streaming while outperforming existing streaming methods in long-term stability. The ultimate test for such a system is its performance over a truly infinite horizon, a capability that has been impossible to rigorously validate due to the lack of extremely long-term, continuous benchmarks. To address this critical gap, we introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames. This provides the definitive evaluation platform for future research in long-term 3D geometry understanding. Code is available at: https://github.com/AutoLab-SAI-SJTU/InfiniteVGGT

InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams

TL;DR

The Long3D benchmark is introduced, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames, and provides the definitive evaluation platform for future research in long-term 3D geometry understanding.

Abstract

The grand vision of enabling persistent, large-scale 3D visual geometry understanding is shackled by the irreconcilable demands of scalability and long-term stability. While offline models like VGGT achieve inspiring geometry capability, their batch-based nature renders them irrelevant for live systems. Streaming architectures, though the intended solution for live operation, have proven inadequate. Existing methods either fail to support truly infinite-horizon inputs or suffer from catastrophic drift over long sequences. We shatter this long-standing dilemma with InfiniteVGGT, a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache. Capitalizing on this, we devise a training-free, attention-agnostic pruning strategy that intelligently discards obsolete information, effectively ``rolling'' the memory forward with each new frame. Fully compatible with FlashAttention, InfiniteVGGT finally alleviates the compromise, enabling infinite-horizon streaming while outperforming existing streaming methods in long-term stability. The ultimate test for such a system is its performance over a truly infinite horizon, a capability that has been impossible to rigorously validate due to the lack of extremely long-term, continuous benchmarks. To address this critical gap, we introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames. This provides the definitive evaluation platform for future research in long-term 3D geometry understanding. Code is available at: https://github.com/AutoLab-SAI-SJTU/InfiniteVGGT
Paper Structure (20 sections, 5 equations, 6 figures, 7 tables)

This paper contains 20 sections, 5 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Classical Offline and Online Reconstruction.
  4. Learning-based Offline 3D Reconstruction.
  5. Method
  6. Preliminaries
  7. From Offline to Online 3D Reconstruction.
  8. Motivation and Analysis
  9. Diversity-aware Rolling Memory
  10. Immutable Anchor Token.
  11. Diversity-quantified Token Retention.
  12. Layer-wise Adaptive Budget Allocation
  13. Long3D Benchmark
  14. Experiments
  15. Experiments Setup
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Paradigm Comparison between previous online and offline 3D geometry understanding and our InfiniteVGGT.
  • Figure 2: Visualization Results.(a) Attention maps from the current frame to adjacent historical cached frames, demonstrating near-identical distributions due to minimal viewpoint shifts in online streaming camera motion. (b) PCA embeddings of query (Q) and key (K) vectors for representative layers and heads, revealing clustering and redundancy in the feature space.
  • Figure 3: Overview of the InfiniteVGGT, illustrating a rolling memory paradigm that prunes KV cache contents to prevent VRAM accumulation over time, employing key cosine similarity and adaptive layer-wise allocation for 3D geometry understanding.
  • Figure 4: Long3D Examples. Views and global point clouds of different scenes.
  • Figure 5: Qualitative Results of 3D Reconstruction.
  • ...and 1 more figures