Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning

Sucheng Ren, Hongru Zhu, Chen Wei, Yijiang Li, Alan Yuille, Cihang Xie

TL;DR

ARVideo introduces a self-supervised video representation learning method that uses autoregressive pretraining over spatiotemporal video token clusters and a randomized prediction order. By clustering tokens into spatial, temporal, or spatiotemporal elements and employing a random rasterization strategy, it captures rich multidimensional context while reducing computation compared to prior methods. On Kinetics-400 and Something-Something V2, ARVideo matches or surpasses VideoMAE while delivering faster training and lower GPU memory usage, and it shows solid transfer to AVA v2.2 and HMDB with competitive performance. Ablation studies confirm the effectiveness of spatiotemporal clustering and randomized ordering, establishing autoregressive video pretraining as a practical alternative to masked video modeling for robust, scalable video representations.

Abstract

This paper presents a new self-supervised video representation learning framework, ARVideo, which autoregressively predicts the next video token in a tailored sequence order. Two key designs are included. First, we organize autoregressive video tokens into clusters that span both spatially and temporally, thereby enabling a richer aggregation of contextual information compared to the standard spatial-only or temporal-only clusters. Second, we adopt a randomized spatiotemporal prediction order to facilitate learning from multi-dimensional data, addressing the limitations of a handcrafted spatial-first or temporal-first sequence order. Extensive experiments establish ARVideo as an effective paradigm for self-supervised video representation learning. For example, when trained with the ViT-B backbone, ARVideo competitively attains 81.2% on Kinetics-400 and 70.9% on Something-Something V2, which are on par with the strong benchmark set by VideoMAE. Importantly, ARVideo also demonstrates higher training efficiency, i.e., it trains 14% faster and requires 58% less GPU memory compared to VideoMAE.

ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning

TL;DR

ARVideo introduces a self-supervised video representation learning method that uses autoregressive pretraining over spatiotemporal video token clusters and a randomized prediction order. By clustering tokens into spatial, temporal, or spatiotemporal elements and employing a random rasterization strategy, it captures rich multidimensional context while reducing computation compared to prior methods. On Kinetics-400 and Something-Something V2, ARVideo matches or surpasses VideoMAE while delivering faster training and lower GPU memory usage, and it shows solid transfer to AVA v2.2 and HMDB with competitive performance. Ablation studies confirm the effectiveness of spatiotemporal clustering and randomized ordering, establishing autoregressive video pretraining as a practical alternative to masked video modeling for robust, scalable video representations.

Abstract

This paper presents a new self-supervised video representation learning framework, ARVideo, which autoregressively predicts the next video token in a tailored sequence order. Two key designs are included. First, we organize autoregressive video tokens into clusters that span both spatially and temporally, thereby enabling a richer aggregation of contextual information compared to the standard spatial-only or temporal-only clusters. Second, we adopt a randomized spatiotemporal prediction order to facilitate learning from multi-dimensional data, addressing the limitations of a handcrafted spatial-first or temporal-first sequence order. Extensive experiments establish ARVideo as an effective paradigm for self-supervised video representation learning. For example, when trained with the ViT-B backbone, ARVideo competitively attains 81.2% on Kinetics-400 and 70.9% on Something-Something V2, which are on par with the strong benchmark set by VideoMAE. Importantly, ARVideo also demonstrates higher training efficiency, i.e., it trains 14% faster and requires 58% less GPU memory compared to VideoMAE.
Paper Structure (32 sections, 2 equations, 3 figures, 10 tables)

This paper contains 32 sections, 2 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Representation Learning
  4. Autoregressive Pretraining
  5. Method
  6. Generative Pretrained Transformer
  7. ImageGPT
  8. ARVideo
  9. Pixel grouping
  10. From Pixels to Video Tokens.
  11. From Tokens to Spatial Clusters.
  12. From Tokens to Temporal Clusters.
  13. From Tokens to Spatiotemporal Clusters.
  14. SpatialTemporal Prediction Order
  15. Pre-defined order.
  16. ...and 17 more sections

Figures (3)

  • Figure 1: ARVideo autoregressive predicts spatiotemporal cluster from grouping tokens span spatial and temporal dimension.
  • Figure 2: Comparison between video token and different cluster.
  • Figure 3: The attention rank comparison between VideoMAE and ARVideo