Generative Pre-trained Autoregressive Diffusion Transformer

TL;DR

GPDiT introduces Generative Pre-trained Autoregressive Diffusion Transformer, a framework that unifies autoregressive sequence modeling with diffusion under a continuous latent space to enable long-range video synthesis. It implements framewise autoregressive diffusion with a lightweight causal attention variant and a parameter-free rotation-based time-conditioning mechanism, improving efficiency while maintaining temporal coherence. Across video generation, representation, and few-shot multitask learning, GPDiT achieves competitive or state-of-the-art results on benchmarks such as MSRVTT and UCF-101, and demonstrates strong zero-shot and few-shot generalization, highlighting its potential as a unified model for video understanding and generation. The work provides practical design choices—causal attention with KV caching, rotation-based time conditioning, and latent-space diffusion—to scale diffusion-transformer video models to long sequences and multi-task scenarios.

Abstract

In this work, we present GPDiT, a Generative Pre-trained Autoregressive Diffusion Transformer that unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis, within a continuous latent space. Instead of predicting discrete tokens, GPDiT autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics and semantic consistency across frames. This continuous autoregressive framework not only enhances generation quality but also endows the model with representation capabilities. Additionally, we introduce a lightweight causal attention variant and a parameter-free rotation-based time-conditioning mechanism, improving both the training and inference efficiency. Extensive experiments demonstrate that GPDiT achieves strong performance in video generation quality, video representation ability, and few-shot learning tasks, highlighting its potential as an effective framework for video modeling in continuous space.

Figures (11)

• Figure 1: Video Generation and Few-Shot Multitask Learning. The left side of the figure illustrates the model's video generation capability: given a set of initial frames, the model can continue the sequence by generating denoised frames. The right side showcases the model's multitask learning ability, similar to the approach presented in radford2019language. After few-shot fine-tuning, the model is capable of performing a variety of tasks, such as translating high-level features to low-level features, converting low-level features to high-level features, and executing style transfer across video sequences.
• Figure 2: Left plane: An overview of GPDiT inference. Middle plane: The architecture of a typical GPDiT block, where adaLN-Zero is replaced with our rotation-based time conditioning, and causal attention is adopted instead of conventional bidirectional attention. Right plane: An illustration of the rotation-based view of the diffusion forward process, where the data and noise components evolve through a parameter-free rotation in the complex plane.
• Figure 3: Illustration of two causal attention variants. Both apply intra-frame full attention and inter-frame causal attention, but differ in cross-frame attention handling between clean frames. $c_i$ and $n_i$ denote clean and noisy frames, respectively.
• Figure 4: Video generation of the subsequent 16 frames conditioned on the initial 13 frames from the MovieGenBenchmark dataset, with the frames sampled at three-frame intervals thereafter. For more details, zoom in to observe finer aspects of the generation process.
• Figure 5: Linear probing performance of GPDiT across different training settings.