Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling

Gongye Liu, Haoze Sun, Jiayi Li, Fei Yin, Yujiu Yang

TL;DR

Inverse problems $y=Hx+n$ are addressed with pre-trained diffusion priors by introducing Shortcut Sampling for Diffusion (SSD) that uses a transitional state $\mathscr{E}$ to bridge the measured image $y$ and the restored image $x$ via the forward process, starting from $H^{\dagger}y$. SSD replaces random-noise initialization with Distortion Adaptive Inversion (DA Inversion) to produce $\mathscr{E}$ and applies back projection as a consistency constraint during generation, with SSD$^+$ for noisy or imperfect degradation. The method achieves competitive or state-of-the-art results at low NFEs (e.g., 30 NFEs) and even surpasses some baselines at 100 NFEs across multiple IR tasks on CelebA and ImageNet. The work provides code and demonstrates speedups and robustness for zero-shot diffusion-based inverse problems.

Abstract

Diffusion models have recently demonstrated an impressive ability to address inverse problems in an unsupervised manner. While existing methods primarily focus on modifying the posterior sampling process, the potential of the forward process remains largely unexplored. In this work, we propose Shortcut Sampling for Diffusion(SSD), a novel approach for solving inverse problems in a zero-shot manner. Instead of initiating from random noise, the core concept of SSD is to find a specific transitional state that bridges the measurement image y and the restored image x. By utilizing the shortcut path of "input - transitional state - output", SSD can achieve precise restoration with fewer steps. To derive the transitional state during the forward process, we introduce Distortion Adaptive Inversion. Moreover, we apply back projection as additional consistency constraints during the generation process. Experimentally, we demonstrate SSD's effectiveness on multiple representative IR tasks. Our method achieves competitive results with only 30 NFEs compared to state-of-the-art zero-shot methods(100 NFEs) and outperforms them with 100 NFEs in certain tasks. Code is available at https://github.com/GongyeLiu/SSD

Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling

TL;DR

Inverse problems are addressed with pre-trained diffusion priors by introducing Shortcut Sampling for Diffusion (SSD) that uses a transitional state to bridge the measured image and the restored image via the forward process, starting from . SSD replaces random-noise initialization with Distortion Adaptive Inversion (DA Inversion) to produce and applies back projection as a consistency constraint during generation, with SSD for noisy or imperfect degradation. The method achieves competitive or state-of-the-art results at low NFEs (e.g., 30 NFEs) and even surpasses some baselines at 100 NFEs across multiple IR tasks on CelebA and ImageNet. The work provides code and demonstrates speedups and robustness for zero-shot diffusion-based inverse problems.

Abstract

Diffusion models have recently demonstrated an impressive ability to address inverse problems in an unsupervised manner. While existing methods primarily focus on modifying the posterior sampling process, the potential of the forward process remains largely unexplored. In this work, we propose Shortcut Sampling for Diffusion(SSD), a novel approach for solving inverse problems in a zero-shot manner. Instead of initiating from random noise, the core concept of SSD is to find a specific transitional state that bridges the measurement image y and the restored image x. By utilizing the shortcut path of "input - transitional state - output", SSD can achieve precise restoration with fewer steps. To derive the transitional state during the forward process, we introduce Distortion Adaptive Inversion. Moreover, we apply back projection as additional consistency constraints during the generation process. Experimentally, we demonstrate SSD's effectiveness on multiple representative IR tasks. Our method achieves competitive results with only 30 NFEs compared to state-of-the-art zero-shot methods(100 NFEs) and outperforms them with 100 NFEs in certain tasks. Code is available at https://github.com/GongyeLiu/SSD
Paper Structure (45 sections, 2 theorems, 47 equations, 19 figures, 5 tables, 1 algorithm)

This paper contains 45 sections, 2 theorems, 47 equations, 19 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Diffusion Models
  4. Denoising Diffusion Probabilistic Models
  5. Denoising Diffusion Implicit Models
  6. Solving inverse problems in a zero-shot way
  7. Method
  8. Shortcut Sampling
  9. Distortion Adaptive Inversion
  10. Why DDIM Inversion Cannot Work Well
  11. Why Original Forward Process Cannot Work Well
  12. Distortion Adaptive Inversion
  13. Back Projection
  14. Expand SSD to noisy IR tasks
  15. Experiments
  16. ...and 30 more sections

Key Result

Theorem 1

Assuming $\epsilon_\theta(x_t, t) \sim \mathcal{N}(\mu, \sigma^2)$, We have: thus: which indicates that after adding random disturbance, $\epsilon_{DA}$ becomes closer to $\mathcal{N}(0, 1)$.

Figures (19)

  • Figure 1: Visual Depiction of "Shortcut Sampling". (a) Previous IR methods initiate from random noise $x_T$, taking unnecessary steps to generate the layout and structure; (b) SSD(ours) modifies the forward process to obtain a better transitional state, employing a shortcut-sampling path of "Input-Transitional State-Target" to restore images with fewer steps.
  • Figure 2: Overview of the proposed SSD. We propose a shortcut sampling pipeline, instead of starting from random noise and spending lots of steps to generate the overall layout and structure, we use Distortion Adaptive Inversion to obtain the transitional state, a noisy image that contains most structure information of the input image. Then during the generation process, we iteratively perform the denoising step and the back projection step to generate images with detailed texture while keeping the restored images consistent with the input images.
  • Figure 3: Comparison of reconstruction results between different Inversion Methods.(a)DDIM Inversion produces faithful but unrealistic results. (b)DDPM Inversion produces realistic but unfaithful results (c)Distortion Adaptive Inversion(ours) produces both realistic and faithful results
  • Figure 4: Qualitative results of different zero-shot IR methods on CelebA and Imagenet Dataset.
  • Figure 5: Colorization results of different zero-shot IR methods on ImageNet Dataset
  • ...and 14 more figures

Theorems & Definitions (5)

  • Definition 1: Distortion Adaptive Inversion
  • Theorem 1
  • Definition 2: Distortion Adaptive Inversion
  • Theorem 2
  • proof