DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models

TL;DR

DiCoDe introduces Diffusion-Compressed Deep Tokens to enable autoregressive video generation with language models by compressing video clips into high-level deep tokens using a diffusion-based tokenizer learned from a video diffusion prior. By treating videos as temporally ordered sequences and modeling token distributions with Gaussian or Gaussian Mixture Model losses, it leverages vanilla autoregressive transformers (e.g., GPT2, Llama3.2) to generate long videos efficiently. The approach achieves substantial compression (approximately 1000$ imes$ reduction in token count) and demonstrates competitive quality on zero-shot benchmarks with scalable performance as AR model size grows, using only a modest compute budget (4 A100 GPUs). This work paves a path toward scalable, long-form video generation via temporal autoregression and transfer of language model capabilities to the video domain. DiCoDe also highlights practical considerations such as tokenization quality, variance modeling, and data limitations, pointing to future work in larger-scale training and more diverse video data.

Abstract

Videos are inherently temporal sequences by their very nature. In this work, we explore the potential of modeling videos in a chronological and scalable manner with autoregressive (AR) language models, inspired by their success in natural language processing. We introduce DiCoDe, a novel approach that leverages Diffusion-Compressed Deep Tokens to generate videos with a language model in an autoregressive manner. Unlike existing methods that employ low-level representations with limited compression rates, DiCoDe utilizes deep tokens with a considerable compression rate (a 1000x reduction in token count). This significant compression is made possible by a tokenizer trained through leveraging the prior knowledge of video diffusion models. Deep tokens enable DiCoDe to employ vanilla AR language models for video generation, akin to translating one visual "language" into another. By treating videos as temporal sequences, DiCoDe fully harnesses the capabilities of language models for autoregressive generation. DiCoDe is scalable using readily available AR architectures, and is capable of generating videos ranging from a few seconds to one minute using only 4 A100 GPUs for training. We evaluate DiCoDe both quantitatively and qualitatively, demonstrating that it performs comparably to existing methods in terms of quality while ensuring efficient training. To showcase its scalability, we release a series of DiCoDe configurations with varying parameter sizes and observe a consistent improvement in performance as the model size increases from 100M to 3B. We believe that DiCoDe's exploration in academia represents a promising initial step toward scalable video modeling with AR language models, paving the way for the development of larger and more powerful video generation models.

Figures (9)

• Figure 1: A VQ-VAE-based encoder yu2021vector represent a 2-second $256\times 256$ video clip consisting of 16 frames with 16,384 low-level tokens. Harvesting the prior knowledge of video diffusion models, DiCoDe compresses the same video clip into 16 high-level deep tokens, achieving a 1000$\times$ reduction in token count. This extremely high compression rate facilitates chronological video modeling with autoregressive language models.
• Figure 2: The overall framework of DiCoDe, which consists of a video diffusion model as the tokenizer to extract highly-compressed deep tokens and an autoregressive language model to predict the sequence of deep tokens through modeling distributions.
• Figure 3: Results of tokenization. A 2-second video clip can be reconstructed effectively from merely 2 frames with 16 deep tokens each, even with object motion (left-top), camera motion (right-top), complex scenes (left-bottom), and emerging entities (right-bottom).
• Figure 4: Zering out deep tokens during tokenization reduces the number of entities in reconstructed frames.
• Figure 5: Comparison of short video generation results. DiCoDe generates more coherent, dynamic and prompt-following videos. On the left, it captures the transition from day to night. On the right, it maintains the appearance of the bird across frames with noticeable motion.