Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space

TL;DR

AdapTok tackles efficient video modeling by introducing a temporally causal, 1D latent space tokenizer with adaptive token budgeting. It combines a 3D patch-based input, a block-causal Transformer encoder/decoder with tail-drop, and a block-causal scorer coupled with IPAL to allocate tokens under a global budget. The approach yields superior reconstruction (lower $FVD$) and stronger video generation on UCF-101 and Kinetics-600, while offering a Pareto-optimal trade-off between token usage and quality. These results enable scalable, token-efficient generative video modeling without requiring additional image data, with potential for broader deployment in streaming and resource-constrained settings.

Abstract

We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling.

Figures (14)

• Figure 1: AdapTok performs adaptive tokenization both temporally and across samples. Left-to-right shows token allocation adapting over time, top-to-bottom presents sample-wise allocation under different token budgets. Blue bars indicate the tokens counts used per block (i.e., 4 frames).
• Figure 2: Overview of the proposed AdapTok framework. (a) Adaptive Tokenizer: composed of a block-causal encoder, a block-wise mask sampler, and a block causal decoder for reconstructing video from adaptively masked latent representations. (b) Attention Masks: block-causal attention patterns used in the encoder, decoder, and scorer. (c) Adaptive Scorer: top-down illustrates the generation of ground-truth scores - duplicated latent tokens $z$ are masked and decoded into videos, and perceptual loss $\mathcal{L}_P$ is computed as the quality scores $s$. Bottom-up shows the scorer predicting block-wise quality scores $\hat{s}$ from continuous latents $z$ and quantized latents $z_q$.
• Figure 3: Comparison of rFVD by Token Length. AdapTok achieves better rFVD with fewer tokens than existing causal tokenizers.
• Figure 4: Ablation on adaptive training and inference.
• Figure 4: Comparison of token allocation strategies. ILP achieves the best performance across all metrics: (a) FVD, (b) PSNR, and (c) LPIPS.