Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations

TL;DR

The paper tackles the need for temporally aware visual pretraining by introducing NExT-Vid, an autoregressive framework that uses masked next-frame prediction to learn robust representations of images and videos. It decouples semantic encoding from target decoding via a context-isolated autoregressive predictor and a conditioned flow-matching decoder, enabling high-quality generation and better semantic localization. Empirical results show state-of-the-art performance among generative pretraining methods across benchmarks like IN1K, K400, SSv2, and Diving48, with scalable gains as model size and data increase. The work demonstrates that challenging objectives, appropriate masking, and stable training dynamics (e.g., EMA reference encoders) are key to extracting strong temporal representations for downstream tasks, while also highlighting trade-offs between generation quality and representation fidelity.

Abstract

Recent advances in pretraining general foundation models have significantly improved performance across diverse downstream tasks. While autoregressive (AR) generative models like GPT have revolutionized NLP, most visual generative pretraining methods still rely on BERT-style masked modeling, which often disregards the temporal information essential for video analysis. The few existing autoregressive visual pretraining methods suffer from issues such as inaccurate semantic localization and poor generation quality, leading to poor semantics. In this work, we propose NExT-Vid, a novel autoregressive visual generative pretraining framework that utilizes masked next-frame prediction to jointly model images and videos. NExT-Vid introduces a context-isolated autoregressive predictor to decouple semantic representation from target decoding, and a conditioned flow-matching decoder to enhance generation quality and diversity. Through context-isolated flow-matching pretraining, our approach achieves strong representations. Extensive experiments on large-scale pretrained models demonstrate that our proposed method consistently outperforms previous generative pretraining methods for visual representation learning via attentive probing in downstream classification.

Figures (7)

• Figure 1: Conceptual illustration of context autoregressive flow-matching pretraining. a) Each frame in the video clip is generated based on masked previous ones. b) The encoder output is isolated to strengthen its semantic representation, and the conditioned flow-matching decoder boosts the generation quality and diversity.
• Figure 2: Overview of the proposed pretraining pipeline. The input video is split into two branches. One branch is masked and fed into the encoder to obtain local representations, while the other branch with the entire sequence is processed by the reference encoder for representation alignment. The autoregressive predictor employs cross-attention to aggregate local features and predict the representations for the next frame. The predicted representations is then aligned with the reference encoder and passed to the flow-matching decoder to generate the VAE latent features of the next frame. The custom attention masks for autoregressive modeling are shown in \ref{['fig:attention']}.
• Figure 3: Custom attention masks for autoregressive modeling. The frame-wise causal mask temporally aggregates video frames, while the autoregressive mask ensures that a frame can only see previous frames. Frame-isolated mask enables individual frame generation, effectively preventing information leakage.
• Figure 4: Scaling curves both on training data amount and model parameters. a) As training data increases, model performance initially grows rapidly before stabilizing, and then improves further after a cool-down stage. b) Increasing model parameters also improves performance, especially from ViT-L (300M) to ViT-H (600M). After the cool-down stage, ViT-G (1100M) significantly outperforms ViT-H.
• Figure 5: Visualization of generation. Top and middle: Masked generation on ImageNet and SSv2, where "ori" denotes the original samples. Bottom: Autoregressive generation based on a seed image from ImageNet.