VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction

TL;DR

VideoAR introduces a scalable Visual Autoregressive framework for video generation by marrying multi-scale next-frame prediction with autoregressive modeling through a 3D causal tokenizer and a Transformer backbone. Key innovations include Multi-scale Temporal RoPE for enhanced spatio-temporal encoding, Cross-Frame Error Correction and Random Frame Mask to mitigate error accumulation, and a staged pretraining pipeline that aligns spatial and temporal representations. Empirically, VideoAR achieves state-of-the-art gFVD among autoregressive models on UCF-101, competitive VBench scores on real-world data, and orders-of-magnitude faster inference than diffusion-based methods, demonstrating a practical, efficient alternative for high-quality, temporally coherent video synthesis. The work also outlines limitations in resolution and horizon length, with clear directions for extending long-horizon generation and improving efficiency through attention-sparse strategies and iterative inference.

Abstract

Recent advances in video generation have been dominated by diffusion and flow-matching models, which produce high-quality results but remain computationally intensive and difficult to scale. In this work, we introduce VideoAR, the first large-scale Visual Autoregressive (VAR) framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling. VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR modeling with causal next-frame prediction, supported by a 3D multi-scale tokenizer that efficiently encodes spatio-temporal dynamics. To improve long-term consistency, we propose Multi-scale Temporal RoPE, Cross-Frame Error Correction, and Random Frame Mask, which collectively mitigate error propagation and stabilize temporal coherence. Our multi-stage pretraining pipeline progressively aligns spatial and temporal learning across increasing resolutions and durations. Empirically, VideoAR achieves new state-of-the-art results among autoregressive models, improving FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps by over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based models an order of magnitude larger. These results demonstrate that VideoAR narrows the performance gap between autoregressive and diffusion paradigms, offering a scalable, efficient, and temporally consistent foundation for future video generation research.

Figures (6)

• Figure 1: VideoAR generates high-fidelity and temporally consistent videos from text prompts.
• Figure 2: Overall framework of VideoAR. Given a text prompt, the video frames are first compressed into a sequence of spatio-temporal tokens via a multi-scale causal 3D tokenizer. Each frame is represented by residual maps at multiple scales, which are autoregressively predicted by a Transformer with block-wise causal masking. The input embeddings combine text tokens, accumulated video features, and scale embeddings, while the proposed Multi-Scale Temporal RoPE encodes temporal, spatial, and scale-aware positional information. Random frame masking is applied during training to mitigate exposure bias and improve long-term consistency. Finally, the multi-scale video decoder reconstructs the video frames from the predicted residuals.
• Figure 3: Our proposed Cross-Frame Error Correction.
• Figure 4: Generation results from our VideoAR-4B on VBench and UCF-101 datasets. Zoom in for clearer visualization.
• Figure 5: Visualization for VideoAR's Image-to-Video and Video-to-Video generation performance. I2V refers to Image-to-Video, purple boxes refers to the given image. V2V shot$N$ refers to N times video-to-video Extension of 4 seconds window.