TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

TL;DR

TTSA3R tackles catastrophic forgetting in streaming 3D reconstruction by introducing a dual-module adaptive update mechanism that uses temporal state evolution (TAUM) and spatial observation alignment (SCUM) to gate updates on a fixed-size persistent state. The method fuses temporal and spatial signals via $M_final = M_temp * M_spat$ and updates $S_t$ accordingly, enabling selective refinement while preserving history. It achieves improved long-sequence stability across video depth, pose estimation, and 3D reconstruction benchmarks, with competitive efficiency and memory footprint (about 5 GB, ~18.5 FPS). This training-free approach narrows the gap between online streaming and offline full-attention methods, offering practical benefits for real-time 3D perception.

Abstract

Streaming recurrent models enable efficient 3D reconstruction by maintaining persistent state representations. However, they suffer from catastrophic memory forgetting over long sequences due to balancing historical information with new observations. Recent methods alleviate this by deriving adaptive signals from attention perspective, but they operate on single dimensions without considering temporal and spatial consistency. To this end, we propose a training-free framework termed TTSA3R that leverages both temporal state evolution and spatial observation quality for adaptive state updates in 3D reconstruction. In particular, we devise a Temporal Adaptive Update Module that regulates update magnitude by analyzing temporal state evolution patterns. Then, a Spatial Contextual Update Module is introduced to localize spatial regions that require updates through observation-state alignment and scene dynamics. These complementary signals are finally fused to determine the state updating strategies. Extensive experiments demonstrate the effectiveness of TTSA3R in diverse 3D tasks. Moreover, our method exhibits only 15% error increase compared to over 200% degradation in baseline models on extended sequences, significantly improving long-term reconstruction stability. Our codes will be available soon.

Figures (9)

• Figure 1: Catastrophic forgetting in streaming 3D reconstruction. Given a sequence of input frames, CUT3R wang2025continuous with uniform updates suffers from severe pose drift and geometric distortions over long sequences (left). TTT3R chen2025ttt3r improves robustness but still exhibits artifacts (middle). Our method alleviates these issues through temporal-spatial adaptive updates to achieve coherent 3D reconstruction with accurate camera poses (right).
• Figure 2: Overview of our framework. Our method performs streaming 3D reconstruction from sequential frames through adaptively updating persistent state. A shared ViT encoder extracts visual tokens from input frames, which interact with the persistent state via ViT decoders to generate candidate states. Meanwhile, TAUM evaluates temporal state evolution across frames and SCUM measures spatial interaction between states and observations. Both modules produce complementary temporal and spatial masks, which are combined to update the persistent state at per-token granularity. Task-specific heads are utilized to predict depth maps, camera poses and pointmaps.
• Figure 3: Illustration of Spatial Context Update Module. Spatial update masks are generated by combining cosine dissimilarity and cross-attention signals through multiplication and sigmoid activation.
• Figure 4: Video depth estimation (long sequences) using metric depth accuracy on Bonn palao2019refun dataset.
• Figure 5: Camera pose estimation (long sequences) on TUM-dynamics sturm2012benchmark (left) and ScanNet dai2017scannet (right) datasets.