Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Prediction Transformers without Recurrence or Convolution

Yujin Tang, Lu Qi, Xiangtai Li, Chao Ma, Ming-Hsuan Yang

TL;DR

PredFormer presents a purely transformer-based approach to video prediction that eliminates recurrence and convolution. By deploying Gated Transformer Blocks with both full and factorized spatiotemporal attention patterns, the method achieves state-of-the-art results across Moving MNIST, Human3.6m, TaxiBJ, and WeatherBench while improving efficiency. Extensive ablations reveal how interleaved spatiotemporal designs, gating mechanisms, and absolute positional encoding contribute to performance and generalization. The work demonstrates that global spatiotemporal modeling with carefully designed transformers can outperform CNN- and RNN-based baselines in real-world prediction tasks, offering a practical, scalable baseline for video forecasting.

Abstract

Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP. Following the significant success of ViT, recent works have integrated ViT into both RNN and CNN frameworks, achieving improved performance. While we appreciate these prior approaches, we raise a fundamental question: Is there a simpler yet more effective solution that can eliminate the high computational cost of RNNs while addressing the limited receptive fields and poor generalization of CNNs? How far can it go with a simple pure transformer model for video prediction? In this paper, we propose PredFormer, a framework entirely based on Gated Transformers. We provide a comprehensive analysis of 3D Attention in the context of video prediction. Extensive experiments demonstrate that PredFormer delivers state-of-the-art performance across four standard benchmarks. The significant improvements in both accuracy and efficiency highlight the potential of PredFormer as a strong baseline for real-world video prediction applications. The source code and trained models will be released at https://github.com/yyyujintang/PredFormer.

Video Prediction Transformers without Recurrence or Convolution

TL;DR

PredFormer presents a purely transformer-based approach to video prediction that eliminates recurrence and convolution. By deploying Gated Transformer Blocks with both full and factorized spatiotemporal attention patterns, the method achieves state-of-the-art results across Moving MNIST, Human3.6m, TaxiBJ, and WeatherBench while improving efficiency. Extensive ablations reveal how interleaved spatiotemporal designs, gating mechanisms, and absolute positional encoding contribute to performance and generalization. The work demonstrates that global spatiotemporal modeling with carefully designed transformers can outperform CNN- and RNN-based baselines in real-world prediction tasks, offering a practical, scalable baseline for video forecasting.

Abstract

Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP. Following the significant success of ViT, recent works have integrated ViT into both RNN and CNN frameworks, achieving improved performance. While we appreciate these prior approaches, we raise a fundamental question: Is there a simpler yet more effective solution that can eliminate the high computational cost of RNNs while addressing the limited receptive fields and poor generalization of CNNs? How far can it go with a simple pure transformer model for video prediction? In this paper, we propose PredFormer, a framework entirely based on Gated Transformers. We provide a comprehensive analysis of 3D Attention in the context of video prediction. Extensive experiments demonstrate that PredFormer delivers state-of-the-art performance across four standard benchmarks. The significant improvements in both accuracy and efficiency highlight the potential of PredFormer as a strong baseline for real-world video prediction applications. The source code and trained models will be released at https://github.com/yyyujintang/PredFormer.
Paper Structure (27 sections, 7 equations, 8 figures, 15 tables)

This paper contains 27 sections, 7 equations, 8 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Pure Transformer Based Architecture
  5. Gated Transformer Block
  6. Variants of PredFormer
  7. Experiments
  8. Synthetic Moving Object Prediction
  9. Real-world Human Motion Prediction
  10. Traffic Flow Prediction
  11. Weather Forecasting
  12. Ablation Study and Discussion
  13. Conclusion
  14. Appendix
  15. Problem Definition
  16. ...and 12 more sections

Figures (8)

  • Figure 1: Main categories of video prediction framework. (a) Recurrent-based Framework (b) CNN Encoder-Decoder-based Recurrent-free Framework. (c) Pure transformer-based Recurrent-free Framework.
  • Figure 2: Overview of the PredFormer framework.
  • Figure 3: (a) Gated Transformer Block (b) Full Attention Encoder and Factorized Encoders (c) Interleaved Encoders with Binary, Triplet, and Quadrupled design
  • Figure 4: Visualizations on Moving MNIST. $\text{Error} = \lvert \text{Prediction}-\text{Target}\rvert$. We amplify the error for better comparison.
  • Figure 5: Visualizations on WeatherBench for global temperature forecasting.
  • ...and 3 more figures