VidTok: A Versatile and Open-Source Video Tokenizer

TL;DR

VidTok tackles the redundancy in video data by providing a versatile, open-source video tokenizer that supports both continuous and discrete latent representations. It introduces a decoupled spatial-temporal architecture with 2D up/downsampling, an AlphaBlender temporal module, and Finite Scalar Quantization to stabilize discrete tokenization. A two-stage training regime with low-resolution pretraining and decoder-only fine-tuning, plus reduced-frame-rate data, yields strong reconstruction across PSNR, SSIM, LPIPS, and FVD on standard benchmarks. The work includes extensive ablations and positions VidTok as a practical foundation for future open-source video modeling research.

Abstract

Encoding video content into compact latent tokens has become a fundamental step in video generation and understanding, driven by the need to address the inherent redundancy in pixel-level representations. Consequently, there is a growing demand for high-performance, open-source video tokenizers as video-centric research gains prominence. We introduce VidTok, a versatile video tokenizer that delivers state-of-the-art performance in both continuous and discrete tokenizations. VidTok incorporates several key advancements over existing approaches: 1) model architecture such as convolutional layers and up/downsampling modules; 2) to address the training instability and codebook collapse commonly associated with conventional Vector Quantization (VQ), we integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3) improved training strategies, including a two-stage training process and the use of reduced frame rates. By integrating these advancements, VidTok achieves substantial improvements over existing methods, demonstrating superior performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD, under standardized evaluation settings.

Figures (6)

• Figure 1: Illustration of the quantitative comparison of discrete and continuous tokenization performance across our VidTok model and state-of-the-art methods, evaluated using four metrics: PSNR, SSIM, LPIPS, and FVD. All performance metrics are obtained through experiments conducted under a consistent evaluation protocol to ensure fairness and comparability. Larger chart areas correspond to better performance across all metrics.
• Figure 2: An overview of video tokenizers.
• Figure 3: The improved model architecture. In the context of a causal setting, consider an input with dimensions $T \times H \times W = 17 \times 256 \times 256$. Assuming a temporal compression factor of $4$ and a spatial compression factor of $8$, the intermediate latent representation is reduced to dimensions $T \times H \times W = 5 \times 32 \times 32$.
• Figure 4: Left: Vector Quantization (VQ) employed in Vector Quantised-Variational AutoEncoder (VQ-VAE) van2017neuralvqvae. Right: Finite Scalar Quantization (FSQ) mentzer2024finite utilized in our model.
• Figure 5: Qualitative comparison with the state-of-the-art video tokenizers.