AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion

TL;DR

AR-Diffusion tackles the challenge of generating temporally coherent, variable-length videos by marrying auto-regressive generation with diffusion in a latent space. It introduces an AR-VAE encoder–decoder to produce compact video tokens and applies a non-decreasing per-frame noise schedule, along with FoPP (training) and AD (inference) schedulers, to stabilize learning and enable flexible generation. The method achieves state-of-the-art or competitive results across four benchmarks, with notable improvements on UCF-101 and strong qualitative motion and detail. This approach offers a scalable path to asynchronous video generation with robust temporal coherence and adjustable inference behavior, suitable for varied-length video synthesis tasks.

Abstract

The task of video generation requires synthesizing visually realistic and temporally coherent video frames. Existing methods primarily use asynchronous auto-regressive models or synchronous diffusion models to address this challenge. However, asynchronous auto-regressive models often suffer from inconsistencies between training and inference, leading to issues such as error accumulation, while synchronous diffusion models are limited by their reliance on rigid sequence length. To address these issues, we introduce Auto-Regressive Diffusion (AR-Diffusion), a novel model that combines the strengths of auto-regressive and diffusion models for flexible, asynchronous video generation. Specifically, our approach leverages diffusion to gradually corrupt video frames in both training and inference, reducing the discrepancy between these phases. Inspired by auto-regressive generation, we incorporate a non-decreasing constraint on the corruption timesteps of individual frames, ensuring that earlier frames remain clearer than subsequent ones. This setup, together with temporal causal attention, enables flexible generation of videos with varying lengths while preserving temporal coherence. In addition, we design two specialized timestep schedulers: the FoPP scheduler for balanced timestep sampling during training, and the AD scheduler for flexible timestep differences during inference, supporting both synchronous and asynchronous generation. Extensive experiments demonstrate the superiority of our proposed method, which achieves competitive and state-of-the-art results across four challenging benchmarks.

Figures (8)

• Figure 1: Different generative models employ different constraints on the timestep compositions and thus exhibit different properties.
• Figure 2: The overall framework of our proposed (a) AR-VAE and (b) AR-Diffusion. AR-VAE encodes videos into latent video tokens and AR-Diffusion models asynchronous video generation in the latent space.
• Figure 3: Settings and properties of different timestep constraints when the number of frames and timesteps is set to be $F=3$ and $T=3$, respectively.
• Figure 4: Qualitative comparison of existing video generative methods and our AR-Diffusion.
• Figure 5: Loss curve of Diffusion Forcing diffusion_forcing on UCF-101.