Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation

TL;DR

This paper proposes an effective method that can be applied to mainstream video diffusion models and achieves high fidelity based on supplementing more precise image information and noise rectification based on supplementing more precise image information and noise rectification.

Abstract

Image-to-video (I2V) generation tasks always suffer from keeping high fidelity in the open domains. Traditional image animation techniques primarily focus on specific domains such as faces or human poses, making them difficult to generalize to open domains. Several recent I2V frameworks based on diffusion models can generate dynamic content for open domain images but fail to maintain fidelity. We found that two main factors of low fidelity are the loss of image details and the noise prediction biases during the denoising process. To this end, we propose an effective method that can be applied to mainstream video diffusion models. This method achieves high fidelity based on supplementing more precise image information and noise rectification. Specifically, given a specified image, our method first adds noise to the input image latent to keep more details, then denoises the noisy latent with proper rectification to alleviate the noise prediction biases. Our method is tuning-free and plug-and-play. The experimental results demonstrate the effectiveness of our approach in improving the fidelity of generated videos. For more image-to-video generated results, please refer to the project website: https://noise-rectification.github.io.

Figures (7)

• Figure 1: The general framework of image diffusion model and video diffusion model with inflated 3D U-Net structure.
• Figure 2: Two basic approaches in existing research and community regarding image-to-video generation.
• Figure 3: The framework of our tuning-free image-to-video method. (a) represents the inference pipeline, where the input image is noised into the initial latent and the predicted noise of the inflated 3D U-Net will be rectified during the denoising process. (b) illustrates the detailed generation process of our method. The visualization of intermediate steps shows our method can effectively refine the denoising direction, making intermediate results closer to the given image.
• Figure 4: Visual comparison with current image-to-video methods. We use AnimateDiff 25_animatediff_guo2023animatediff as the VLDM. "Ctrl.R.O." means ControlNet Reference-Only method 12_ControlNet_zhang2023adding. Our method achieves higher fidelity in the video sequences with the given image.
• Figure 5: Ablation study on the weight of noise rectification. We fix the rectification timestep $\tau$ and change the rectification weights $\omega$ for different frames.