Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model

Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, Jun Zhu

TL;DR

This paper proposes to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap.

Abstract

Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.

Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model

TL;DR

This paper proposes to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap.

Abstract

Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.
Paper Structure (31 sections, 1 theorem, 16 equations, 13 figures, 9 tables, 2 algorithms)

This paper contains 31 sections, 1 theorem, 16 equations, 13 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion Models.
  4. Diffusion Models for Image-to-Video Generation.
  5. Method
  6. Identifying Conditional Image Leakage in Image-to-video Diffusion Models
  7. Solving Conditional Image Leakage in Image-to-video Diffusion Models
  8. Inference strategy.
  9. Training strategy.
  10. Understanding Existing Work from Conditional Image Leakage
  11. Experiments
  12. Setup
  13. Datasets.
  14. Evaluation Metrics.
  15. Implementation Details.
  16. ...and 16 more sections

Key Result

Proposition 1

Given a normal distribution $p_M(X_M) = {\mathcal{N}}(X_M;{\bm{\mu}}_p,\sigma_p^2{\bm{I}})$ and $q_M(X_M)$ is the margin distribution of diffusion forward process at time $M$, with the forward trainsition kernel $q_{M|0}(X_M|X_0)={\mathcal{N}}(X_M;\alpha_M X_0, \sigma_M^2 {\bm{I}})$, the minimizatio where $q(X_0)$ denotes the data distribution , $d$ denotes the dimension of the data, and $X_0^{(j)

Figures (13)

  • Figure 1: The issue of existing I2V-DMs. Regardless of input motion scores (Input MS), the output motion scores (Output MS) are consistently lower than expected. In contrast, our method yields output motion scores either higher or lower than Input MS with reduced error.
  • Figure 2: Identifying conditional image leakage. As time step progresses, the noisy input becomes heavily corrupted, whereas the conditional image retains considerable detail from GT. This biases the model to over-rely on the conditional image at large $t$, resulting in videos with less motion than GT.
  • Figure 3: Benefits of Analytic-Init. (a) An early start time $M$ enhances motion but a too-small $M$ degrades visual quality due to the training-inference gap, which Analytic-Init helps to reduce. (b) Analytic-Init produces higher motion scores with lower errors, mitigating conditional image leakage.
  • Figure 4: Visualization of TimeNoise and the impact of tuning its hyperparameters. (a) The designed $p_t(\beta_s)$ favors high noise levels at large $t$, gradually shifting to lower noise levels as $t$ decreases. This is achieved by (b) $\mu(t)$ increasing monotonically with $t$. Finally, (c) modifying $a$ and $\beta_m$ enables a trade-off between dynamic motion and image alignment.
  • Figure 4: Training settings for DynamiCrafter xing2023dynamicrafter and VideoCrafter1 chen2023videocrafter1.
  • ...and 8 more figures

Theorems & Definitions (1)

  • Proposition 1