VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate

TL;DR

The paper tackles the computational bottleneck of diffusion-transformer video generation by leveraging temporal non-uniformity in real-world content. It introduces VGDFR, a training-free approach that dynamically adjusts latent frame rates during denoising via a compression module and latent-space frame merging, complemented by a Dynamic Rotary Position Embedding (DyRoPE) strategy. Experiments on HunyuanVideo DiT demonstrate 1.2×–3× speedups with minimal quality loss and no fine-tuning required, validating both the practicality and robustness of the method. Ablation studies and RoPE analyses further show that frame merging and layer-wise embedding adaptation are key to maintaining semantic fidelity. Overall, VGDFR offers a scalable, practical acceleration for high-resolution video diffusion with preserved content and temporal coherence.

Abstract

Diffusion Transformer(DiT)-based generation models have achieved remarkable success in video generation. However, their inherent computational demands pose significant efficiency challenges. In this paper, we exploit the inherent temporal non-uniformity of real-world videos and observe that videos exhibit dynamic information density, with high-motion segments demanding greater detail preservation than static scenes. Inspired by this temporal non-uniformity, we propose VGDFR, a training-free approach for Diffusion-based Video Generation with Dynamic Latent Frame Rate. VGDFR adaptively adjusts the number of elements in latent space based on the motion frequency of the latent space content, using fewer tokens for low-frequency segments while preserving detail in high-frequency segments. Specifically, our key contributions are: (1) A dynamic frame rate scheduler for DiT video generation that adaptively assigns frame rates for video segments. (2) A novel latent-space frame merging method to align latent representations with their denoised counterparts before merging those redundant in low-resolution space. (3) A preference analysis of Rotary Positional Embeddings (RoPE) across DiT layers, informing a tailored RoPE strategy optimized for semantic and local information capture. Experiments show that VGDFR can achieve a speedup up to 3x for video generation with minimal quality degradation.

Figures (4)

• Figure 1: VGDFR: A training-free approach that accelerates video generation by dynamically determines the frame rate based on content already generated in denoising steps. By introducing VGDFR Compression Module, we apply various compression ratio of different time segments in order to reduce tokens for decreasing the computation cost of following DiT inference.
• Figure 2: VGDFR Compression Module dynamically adjusts the frame rate for different time segments in a video to reduce the number of tokens needed for DiT inference.
• Figure 3: Visualization of Text-to-Video Generation on VBench Prompt Set. Three rows of the figure displays frames extracted from the start, middle, and latter of the generated videos. The leftmost column shows videos generated by the original Hunyuan Video model, while the subsequent columns present videos generated by VGDFR under different parameter configurations. Here, $\theta$ denotes the threshold, and $k$ represents the denoising timestep at which dynamic frame rate compression is initiated.
• Figure 4: Contribution of components. Left column: Video generated by VGDFR. Middle two column: Video generated by VGDFR with Compression Module completely removed. Right column: Video generated by omitting denoising and renoising operations in the VGDFR Compression Module.