Rethinking Video Tokenization: A Conditioned Diffusion-based Approach

TL;DR

This work tackles the instability and training complexity of GAN-based decoders in video tokenizers by introducing CDT, a conditioned diffusion-based video tokenizer. CDT uses a causal 3D encoder to produce latent representations that condition a reverse diffusion process, trained with a simple combination of diffusion loss, KL regularization, and LPIPS perceptual loss, and accelerated at inference via DDIM with a feature cache for arbitrarily long videos. The key contributions include the first diffusion-based video tokenizer without multi-stage GAN tricks, single-step sampling achieving state-of-the-art reconstruction, and a competitive, efficient latent video generation capability. The approach promises more stable training, high fidelity reconstructions, and practical efficiency for long videos, with code and pretrained weights openly available.

Abstract

Existing video tokenizers typically use the traditional Variational Autoencoder (VAE) architecture for video compression and reconstruction. However, to achieve good performance, its training process often relies on complex multi-stage training tricks that go beyond basic reconstruction loss and KL regularization. Among these tricks, the most challenging is the precise tuning of adversarial training with additional Generative Adversarial Networks (GANs) in the final stage, which can hinder stable convergence. In contrast to GANs, diffusion models offer more stable training processes and can generate higher-quality results. Inspired by these advantages, we propose CDT, a novel Conditioned Diffusion-based video Tokenizer, that replaces the GAN-based decoder with a conditional causal diffusion model. The encoder compresses spatio-temporal information into compact latents, while the decoder reconstructs videos through a reverse diffusion process conditioned on these latents. During inference, we incorporate a feature cache mechanism to generate videos of arbitrary length while maintaining temporal continuity and adopt sampling acceleration technique to enhance efficiency. Trained using only a basic MSE diffusion loss for reconstruction, along with KL term and LPIPS perceptual loss from scratch, extensive experiments demonstrate that CDT achieves state-of-the-art performance in video reconstruction tasks with just a single-step sampling. Even a scaled-down version of CDT (3$\times$ inference speedup) still performs comparably with top baselines. Moreover, the latent video generation model trained with CDT also exhibits superior performance. The source code and pretrained weights are available at https://github.com/ali-vilab/CDT.

Figures (13)

• Figure 1: Overview: CDT adopts a causal diffusion-based decoder, achieving reconstruction via a reverse diffusion process that is conditioned on the latents extracted by encoder.
• Figure 2: The architecture of the proposed CDT.
• Figure 2: Comparison results of the reconstruction effiency.
• Figure 3: Comparison of FVD trends.
• Figure 4: Examples of videos generated by Latte using CDT.