Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training

Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, Yueqi Duan

Abstract

Humans perceive and understand real-world spaces through a stream of visual observations. Therefore, the ability to streamingly maintain and update spatial evidence from potentially unbounded video streams is essential for spatial intelligence. The core challenge is not simply longer context windows but how spatial information is selected, organized, and retained over time. In this paper, we propose Spatial-TTT towards streaming visual-based spatial intelligence with test-time training (TTT), which adapts a subset of parameters (fast weights) to capture and organize spatial evidence over long-horizon scene videos. Specifically, we design a hybrid architecture and adopt large-chunk updates parallel with sliding-window attention for efficient spatial video processing. To further promote spatial awareness, we introduce a spatial-predictive mechanism applied to TTT layers with 3D spatiotemporal convolution, which encourages the model to capture geometric correspondence and temporal continuity across frames. Beyond architecture design, we construct a dataset with dense 3D spatial descriptions, which guides the model to update its fast weights to memorize and organize global 3D spatial signals in a structured manner. Extensive experiments demonstrate that Spatial-TTT improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks. Project page: https://liuff19.github.io/Spatial-TTT.

Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training

Abstract

Humans perceive and understand real-world spaces through a stream of visual observations. Therefore, the ability to streamingly maintain and update spatial evidence from potentially unbounded video streams is essential for spatial intelligence. The core challenge is not simply longer context windows but how spatial information is selected, organized, and retained over time. In this paper, we propose Spatial-TTT towards streaming visual-based spatial intelligence with test-time training (TTT), which adapts a subset of parameters (fast weights) to capture and organize spatial evidence over long-horizon scene videos. Specifically, we design a hybrid architecture and adopt large-chunk updates parallel with sliding-window attention for efficient spatial video processing. To further promote spatial awareness, we introduce a spatial-predictive mechanism applied to TTT layers with 3D spatiotemporal convolution, which encourages the model to capture geometric correspondence and temporal continuity across frames. Beyond architecture design, we construct a dataset with dense 3D spatial descriptions, which guides the model to update its fast weights to memorize and organize global 3D spatial signals in a structured manner. Extensive experiments demonstrate that Spatial-TTT improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks. Project page: https://liuff19.github.io/Spatial-TTT.
Paper Structure (23 sections, 4 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 23 sections, 4 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual-based Spatial Intelligence
  4. Test-Time Training
  5. Spatial-TTT
  6. Preliminary
  7. Overall Framework
  8. Bridging Sparse Spatial QA with Dense Supervision
  9. Spatial-Aware Progressive Training
  10. Experiments
  11. Experiment Setup
  12. General Spatial Understanding Results
  13. Streaming Spatial Sensing Results
  14. Ablation Study and Analysis
  15. Conclusion
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Spatial-TTT. Given a Visual-based Spatial task, our proposed Spatial-TTT updates spatial state with streaming chunks then answers the question.
  • Figure 2: Overview of Spatial-TTT. The model employs a hybrid architecture that interleaves TTT layers with self-attention anchor layers at a 3:1 ratio to preserve pretrained knowledge while enabling efficient long spatial-context compression. Within each TTT layer, sliding window attention (SWA) and TTT branch operate in parallel with shared Q/K/V projections, where the TTT branch applies spatial predictive mechanism with depthwise spatiotemporal convolution to capture geometric structure and temporal continuity.
  • Figure 3: QA Example Visualization on VSI-Bench
  • Figure 4: QA Example Visualization on VSI-SUPER-RECALL
  • Figure 5: QA Example Visualization on VSI-SUPER-COUNT
  • ...and 1 more figures