REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder

TL;DR

REGEN rethinks video tokenization by replacing encoder-decoder with an encoder-generator framework that uses a diffusion-transformer decoder conditioned on compact latents. The key innovation is a latent conditioning module that produces content-aware positional embeddings, enabling the DiT decoder to generalize across resolutions and achieve up to $32\times$ temporal compression while maintaining plausible reconstructions. Empirical results show superior reconstruction quality at high compression compared to state-of-the-art embedders and demonstrate the latent space's effectiveness for text-to-video generation with a 5B DiT latent-diffusion model, offering substantial efficiency gains in training and inference. This approach has practical implications for scalable video generative modeling and downstream diffusion-based video synthesis tasks.

Abstract

We present a novel perspective on learning video embedders for generative modeling: rather than requiring an exact reproduction of an input video, an effective embedder should focus on synthesizing visually plausible reconstructions. This relaxed criterion enables substantial improvements in compression ratios without compromising the quality of downstream generative models. Specifically, we propose replacing the conventional encoder-decoder video embedder with an encoder-generator framework that employs a diffusion transformer (DiT) to synthesize missing details from a compact latent space. Therein, we develop a dedicated latent conditioning module to condition the DiT decoder on the encoded video latent embedding. Our experiments demonstrate that our approach enables superior encoding-decoding performance compared to state-of-the-art methods, particularly as the compression ratio increases. To demonstrate the efficacy of our approach, we report results from our video embedders achieving a temporal compression ratio of up to 32x (8x higher than leading video embedders) and validate the robustness of this ultra-compact latent space for text-to-video generation, providing a significant efficiency boost in latent diffusion model training and inference.

Figures (15)

• Figure 1: Reconstruction (top) and text-to-video (T2V) generations (bottom) results at 32$\times$ temporal compression. This figure contains video results (for T2V results), best viewed with Adobe Acrobat Reader. Colored bounding boxes in reconstruction results denote the regions with the most difference where MAGVIT-v2 leads to clear artifacts in the tuning fork.
• Figure 2: Overall framework. Our spatiotemporal video encoder $E\left ( \cdot \right )$ encodes the input video $x_{input}$ into two latent frames, content and motion $\left ( z_{c}, z_{m} \right )$. They are processed by the latent expansion module $C_{e}$ and serve as conditioning for the generative decoder.
• Figure 3: Latent conditioning module $C_{e}$. The SIREN network $M_{t}$ maps the time coordinate $t_{f}$ to a feature vector modulated by the motion latent $z_{m}$. The resulting feature is concatenated with the feature value of $z_c$ at the corresponding spatial coordinate $(x,y)$. The concatenated feature is mapped into the DiT hidden dimension by the projector $M_{s}$. We utilize the first frame prediction from SIREN to replace the first frame of expanded $z_{c}$ to ensure consistent representation for both image and video inputs.
• Figure 4: Effectiveness of REGEN at high temporal compression. MAGVIT-v2 suffers from strong temporal artifacts in regions of high motion, such as the dog's face (left) and the toy (right). Areas enclosed in boxes show regions of maximum difference.
• Figure 5: Effectiveness of REGEN at base 4$\times$ temporal compression. Current video embedders suffer from ghosting artifacts for videos with large motion, especially in faces (last row). REGEN performs well and retains plausible spatiotemporal structures from the input.