DeRA: Decoupled Representation Alignment for Video Tokenization

TL;DR

DeRA introduces a decoupled 1D video tokenizer that splits video encoding into appearance and motion streams, each guided by separate foundation-model representations. A Symmetric Alignment-Conflict Projection (SACP) module mitigates gradient conflicts during joint alignment, stabilizing training. Empirically, DeRA achieves superior reconstruction with fewer tokens and sets new state-of-the-art results in class-conditional UCF-101 generation and Kinetics-600 frame prediction, outperforming prior 1D/video tokenizers by a noticeable margin. The approach offers improved efficiency and quality for autoregressive video generation and provides a pathway toward scalable, semantically rich video tokenization.

Abstract

This paper presents DeRA, a novel 1D video tokenizer that decouples the spatial-temporal representation learning in video tokenization to achieve better training efficiency and performance. Specifically, DeRA maintains a compact 1D latent space while factorizing video encoding into appearance and motion streams, which are aligned with pretrained vision foundation models to capture the spatial semantics and temporal dynamics in videos separately. To address the gradient conflicts introduced by the heterogeneous supervision, we further propose the Symmetric Alignment-Conflict Projection (SACP) module that proactively reformulates gradients by suppressing the components along conflicting directions. Extensive experiments demonstrate that DeRA outperforms LARP, the previous state-of-the-art video tokenizer by 25% on UCF-101 in terms of rFVD. Moreover, using DeRA for autoregressive video generation, we also achieve new state-of-the-art results on both UCF-101 class-conditional generation and K600 frame prediction.

Figures (7)

• Figure 1: Comparison of video tokenization approaches. (a) Grid-based tokenizer; (b) Query-based 1D tokenizer; (c) DeRA aligns decoupled representations to the corresponding foundation model (FM), yielding 2$\times$ faster convergence and better performance.
• Figure 2: Overview of our method. DeRA decouples video into the appearance stream and motion stream via a shared encoder. During training, we align appearance and motion latents to the frozen image and video foundation models, respectively. The Symmetric Alignment-Conflict Projection (SACP) module reformulates the conflicting gradients before updating. The two latents are concatenated and quantized to produce discrete tokens that drive the decoder for reconstruction. During inference, external models and regularizations are removed.
• Figure 3: The reconstruction and generation comparisons using different token budgets.
• Figure 4: Comparison of video reconstruction results between LARP wang2024larp and our method on UCF-101 soomro2012dataset dataset.
• Figure 5: Class-conditional video generation comparison on UCF-101 soomro2012dataset dataset. Left: LARP wang2024larp; right: Ours.