Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation

TL;DR

This work tackles the high computational cost of diffusion-based video generation by exploiting inter-frame motion consistency in latent space to reuse coarse-grained denoising across frames. It introduces Dr. Mo, comprising a Motion Transformation Network (MTN) and a Denoising Step Selector (DSS), which learn multi-scale inter-frame motion and dynamically determine the optimal transition point from motion-based propagation to full denoising. Empirical results on UCF-101 and MSR-VTT show that Dr. Mo delivers substantial speedups (approximately 4× over Latent-Shift and 1.5× over SimDA/LaVie) while maintaining or improving perceptual and temporal fidelity (FVD and CLIPSIM). The method also supports video editing via motion-informed style transfer, highlighting its practical impact for efficient video generation and editing in diffusion-based frameworks.

Abstract

Video generation using diffusion-based models is constrained by high computational costs due to the frame-wise iterative diffusion process. This work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent video generation. Our key discovery is that coarse-grained noises in earlier denoising steps have demonstrated high motion consistency across consecutive video frames. Following this observation, Dr. Mo propagates those coarse-grained noises onto the next frame by incorporating carefully designed, lightweight inter-frame motions, eliminating massive computational redundancy in frame-wise diffusion models. The more sensitive and fine-grained noises are still acquired via later denoising steps, which can be essential to retain visual qualities. As such, deciding which intermediate steps should switch from motion-based propagations to denoising can be a crucial problem and a key tradeoff between efficiency and quality. Dr. Mo employs a meta-network named Denoising Step Selector (DSS) to dynamically determine desirable intermediate steps across video frames. Extensive evaluations on video generation and editing tasks have shown that Dr. Mo can substantially accelerate diffusion models in video tasks with improved visual qualities.

Figures (9)

• Figure 1: Left: The spectrum illustrates an increase in high-frequency signals during the denoising process, from steps 900 to 100. Right: High NMI scores between steps 800 and 200 indicate consistent motion dynamics of video frames 0-4 (and 0-8) throughout the denoising process.
• Figure 2: Motion visualization at step 200 accurately captures the movement trends of patch features. At this step, the motion dynamics show consistency with low transformation errors, indicating the potential for reusing steps between step 1000 and 200.
• Figure 3: Dr. Mo consists of two main components: the Motion Transformation Network (MTN) and Denoising Step Selector (DSS). MTN learns motion matrices from semantic latents extracted from U-Net. The DSS is a meta-network that determines the appropriate transition step (denoted as $t^\ast$) for switching from motion-based propagations to denoising. After the transition step, those latent noise is processed by the rest of the diffusion model for video generation.
• Figure 4: Comparison with Latent-Shift using video frames with 256$\times$256 resolution on UCF-101.
• Figure 5: Generated videos with 512$\times$512 resolution.

Theorems & Definitions (1)

• Definition 1: Step-wise Temporal Consistency of Motion Dynamics