MTC-VAE: Multi-Level Temporal Compression with Content Awareness

TL;DR

MTC-VAE introduces multi-level temporal compression for latent video diffusion, enabling content-aware adaptation of per-segment temporal resolution through a Video Clipper and a keyframe-based decoding mechanism. The two-stage training with a flow-guided consistency loss and an alpha-weighted quality-compression objective enables dynamic compression decisions that preserve reconstruction quality while significantly increasing compression rates. When integrated with diffusion transformers like DiT, MTC-VAE accelerates generation and supports longer videos with limited additional compute, achieving up to 92.4% compression improvements with minimal PSNR/SSIM degradation. The approach is validated across multiple open-video benchmarks, ablations, and DiT fine-tuning scenarios, demonstrating practical impact for scalable, high-resolution video synthesis and a meaningful reduction in computational cost.

Abstract

Latent Video Diffusion Models (LVDMs) rely on Variational Autoencoders (VAEs) to compress videos into compact latent representations. For continuous Variational Autoencoders (VAEs), achieving higher compression rates is desirable; yet, the efficiency notably declines when extra sampling layers are added without expanding the dimensions of hidden channels. In this paper, we present a technique to convert fixed compression rate VAEs into models that support multi-level temporal compression, providing a straightforward and minimal fine-tuning approach to counteract performance decline at elevated compression rates.Moreover, we examine how varying compression levels impact model performance over video segments with diverse characteristics, offering empirical evidence on the effectiveness of our proposed approach. We also investigate the integration of our multi-level temporal compression VAE with diffusion-based generative models, DiT, highlighting successful concurrent training and compatibility within these frameworks. This investigation illustrates the potential uses of multi-level temporal compression.

Figures (5)

• Figure 1: Our MTC-VAE model is illustrated. This architecture incorporates a compression algorithm that adaptively manages temporal compression rates$C$ across video segments, deciding the quantity of sampling layers to implement. Only the temporal sample layers are shown, with newly introduced sample layers initialized from existing pretrained ones. The segment area is indicated by the embedding $f_c$. Our linear keyframe predictor is adept at identifying the labeled latent features from the combined latent representation, and segments are divided based on these features. These latent segments are then decoded to generate the final video output. The specific process can be referenced in the pseudocode in Algorithm \ref{['alg:vae']}.
• Figure 2: Using our compression algorithm, we clearly observe that it tends to apply higher compression rates to simpler and slower videos, while employing lower temporal compression rates for more complex or fast-moving videos.
• Figure 3: MTC-VAE enables a finely-tuned DiT to allocate less latent space for simple data with high compression, while dedicating more for complex frames with lower compression rates. We illustrate four videos generated with differing compression: 256× (4×8×8), 512× (8×8×8), and 1024× (16×8×8). This method assists DiT in creating videos with variable frame lengths. The generation process includes producing 720p videos in 50 sampling steps, utilizing carefully designed prompts to emphasize temporal scene transitions. Additional visuals and comparisons can be found in the supplementary \ref{['fig:appearance']} and material.
• Figure 4: Scatter plot illustrates when the compression of 256×(4×8×8) provides equivalent or higher quality compared to 128×(2×8×8). Orange indicates cases where $|\text{PSNR}_{128\times} - \text{PSNR}_{256\times}| \leq 0.5$ or $\text{PSNR}_{256\times} > \text{PSNR}_{128\times}$ for a video segment, whereas blue signifies $\text{PSNR}_{256\times} < (\text{PSNR}_{128\times} - 0.5)$. PSNR Std. denotes the standard deviation of PSNR values across compression levels 128×(2×8×8), 256×(4×8×8), 512×(8×8×8), and 1024×(16×8×8). Optical Flow refers to motion dynamics, and MPEG-4 refers to the level of storage compression applied via the MPEG technique for the video. This data is gathered from 1,024 videos within the Pixabay collection, included in the OpenSoraPlan open-source dataset pku_yuan_lab_and_tuzhan_ai_etc_2024_10948109.
• Figure 5: Visual comparison of our MTC-VAE model's performance across different scenes. Each row shows consistent style preservation despite varying compression rates, demonstrating robust visual quality maintenance across diverse content types.