Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction

TL;DR

The paper tackles the slow sampling inherent in conditional diffusion models for inverse problems by introducing CCDF, a two-stage scheme that starts reverse diffusion from a forward-diffused initialization and interleaves data-consistency steps. Guided by contraction theory, CCDF guarantees exponential convergence and reduces the required reverse steps, especially when paired with a better initial estimate (e.g., NN-based). The approach yields state-of-the-art or competitive results in super-resolution, inpainting, and MRI reconstruction while using far fewer diffusion steps, with practical acceleration demonstrated across datasets. This hybrid method couples fast feed-forward priors with diffusion-based priors, enabling faster, more stable reconstructions suitable for potential real-time deployment.

Abstract

Diffusion models have recently attained significant interest within the community owing to their strong performance as generative models. Furthermore, its application to inverse problems have demonstrated state-of-the-art performance. Unfortunately, diffusion models have a critical downside - they are inherently slow to sample from, needing few thousand steps of iteration to generate images from pure Gaussian noise. In this work, we show that starting from Gaussian noise is unnecessary. Instead, starting from a single forward diffusion with better initialization significantly reduces the number of sampling steps in the reverse conditional diffusion. This phenomenon is formally explained by the contraction theory of the stochastic difference equations like our conditional diffusion strategy - the alternating applications of reverse diffusion followed by a non-expansive data consistency step. The new sampling strategy, dubbed Come-Closer-Diffuse-Faster (CCDF), also reveals a new insight on how the existing feed-forward neural network approaches for inverse problems can be synergistically combined with the diffusion models. Experimental results with super-resolution, image inpainting, and compressed sensing MRI demonstrate that our method can achieve state-of-the-art reconstruction performance at significantly reduced sampling steps.

Figures (17)

• Figure 1: Reconstruction results of three different tasks - super-resolution, inpainting, and MRI reconstruction. Numbers in parenthesis indicate the iteration numbers for reverse diffusion. Proposed method is compared with canonical conditional diffusion models for each task. (a) Corrupted measurement, (b) ILVR choi2021ilvr, score-SDE song2020score, and score-MRI chung2021score, respectively, for each task. (c) Proposed method.
• Figure 2: Plot of average error $\bar{\varepsilon}$ vs. time $t$, using different approaches. (a) Conditional diffusion starts from Gaussian noise ${\bm{x}\xspace}(t)$ and uses full reverse diffusion. (b) CCDF with vanilla initialization: Corrupted data is forward-diffused with a single step up to $t = t_0$, and reverse diffused. (c) CCDF with NN initialization: Initialization with reconstruction from pre-trained NN lets us use much smaller timestep $t = t'_0 < t_0$, and hence faster reverse diffusion.
• Figure 3: Stability of convergence depending on the choice of initialization. (a) Random initialization, large $\varepsilon_0$, (b) vanilla initialization, moderate $\varepsilon_0$, (c) NN initialization, small $\varepsilon_0$.
• Figure 4: Results of super-resolution on AFHQ 256$\times$256 data. First, second and third row denote $\times 4$ SR, $\times 8$ SR, and $\times 16$ SR, respectively. (a) LR input, (b) Ground Truth, (c) ESRGAN wang2018esrgan, (d) SR3 saharia2021image with 20 diffusion steps ($N = 20, \Delta t = 0.05$), (e) ILVR choi2021ilvr with 20 diffusion steps ($N = 20, \Delta t = 0.05$), (f) proposed method with 20 diffusion steps ($N = 100, t_0 = 0.2$).
• Figure 5: Comparison of FID score on $\times$8 SR task. For ILVR, re-scheduling method of IDDPM nichol2021improved was used starting from $T=1$. For CCDF, the step size for discretization is $\Delta t=0.01$ so that the starting point for the reverse diffusion is $t_0 = \Delta t \times \hbox{[number of iteration]}$.

Theorems & Definitions (11)

• Lemma 1
• Theorem 1
• Theorem 2: Shortcut path
• Definition 1: Contraction on ${\mathbb R}^n$
• Theorem A.1
• Corollary 1
• proof
• Lemma A.1
• proof
• Lemma C.1