CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion

TL;DR

CoNo tackles the challenge of tuning-free long video diffusion by introducing two key innovations: a look-back mechanism that inserts an internal noise-prediction stage between two video-extending steps with noise-shuffle strategies, and a long-term consistency regularization that minimizes pixel-wise differences in predicted noises across adjacent clips. The approach explicitly constrains the denoising trajectory to maintain content consistency and smooth transitions when extending videos, enabling longer sequences without retraining. Empirical results demonstrate state-of-the-art scene consistency and perceptual quality under both single- and multi-prompt conditions, with extensive ablations validating the contributions. This work offers a practical, resource-efficient path to high-quality long video generation using existing short-video diffusion models, with potential for broad applicability and further improvements through prompt engineering and model generalization.

Abstract

Tuning-free long video diffusion has been proposed to generate extended-duration videos with enriched content by reusing the knowledge from pre-trained short video diffusion model without retraining. However, most works overlook the fine-grained long-term video consistency modeling, resulting in limited scene consistency (i.e., unreasonable object or background transitions), especially with multiple text inputs. To mitigate this, we propose the Consistency Noise Injection, dubbed CoNo, which introduces the "look-back" mechanism to enhance the fine-grained scene transition between different video clips, and designs the long-term consistency regularization to eliminate the content shifts when extending video contents through noise prediction. In particular, the "look-back" mechanism breaks the noise scheduling process into three essential parts, where one internal noise prediction part is injected into two video-extending parts, intending to achieve a fine-grained transition between two video clips. The long-term consistency regularization focuses on explicitly minimizing the pixel-wise distance between the predicted noises of the extended video clip and the original one, thereby preventing abrupt scene transitions. Extensive experiments have shown the effectiveness of the above strategies by performing long-video generation under both single- and multi-text prompt conditions. The project has been available in https://wxrui182.github.io/CoNo.github.io/.

Figures (16)

• Figure 1: Illustration of the CoNo framework. We propose a "look-back" mechanism that inserts an internal noise prediction stage between two video extending stages to enhance scene consistency. To achieve this, we design the extending and internal initial noise shuffles and constrain the denoising trajectory using selected predicted noise (denoted as $\left[s\right]$ in the figure). Additionally, we apply long-term consistency regularization between adjacent video clips to avoid abrupt content shifts. We obtain the final video by concatenating the frames marked with yellow boxes from different stages.
• Figure 2: Observed content shifts and improvements brought by the proposed Long-term Consistency Regularization.
• Figure 3: To constrain the denoising trajectory of selected frames, we design extending and internal noise shuffles for the initial noise. Different colored blocks represent different video frames, with $z_T$ indicating the initial noise and frame numbers annotated in the top right corner.
• Figure 4: Qualitative comparisons of single-prompt longer video generation.
• Figure 5: Qualitative comparisons of multi-prompt longer video generation.