Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Jiawei Zhang, Jiaxin Zhuang, Cheng Jin, Gen Li, Yuantao Gu

TL;DR

The paper addresses inverse problems by exploiting the denoising capability of pre-trained diffusion priors within a principled optimization framework. It introduces ProjDiff, a two-variable ELBO leveraging an auxiliary variable x_{t_a} and solves the resulting constrained objective with projection gradient descent and gradient truncation. The approach demonstrates competitive or superior performance across linear and nonlinear inverse problems, including image restoration and music source separation, highlighting practical potential. The method offers a flexible, diffusion-based alternative to MAP/plug-and-play strategies with extensions to nonlinear observations and weak-observation regimes through techniques like Restricted Encoding.

Abstract

The recent emergence of diffusion models has significantly advanced the precision of learnable priors, presenting innovative avenues for addressing inverse problems. Since inverse problems inherently entail maximum a posteriori estimation, previous works have endeavored to integrate diffusion priors into the optimization frameworks. However, prevailing optimization-based inverse algorithms primarily exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization task by introducing an auxiliary optimization variable. By employing gradient truncation, the projection gradient descent method is efficiently utilized to solve the corresponding optimization problem. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of a pre-trained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting its potential for practical applications. Code is available at https://github.com/weigerzan/ProjDiff/.

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

TL;DR

The paper addresses inverse problems by exploiting the denoising capability of pre-trained diffusion priors within a principled optimization framework. It introduces ProjDiff, a two-variable ELBO leveraging an auxiliary variable x_{t_a} and solves the resulting constrained objective with projection gradient descent and gradient truncation. The approach demonstrates competitive or superior performance across linear and nonlinear inverse problems, including image restoration and music source separation, highlighting practical potential. The method offers a flexible, diffusion-based alternative to MAP/plug-and-play strategies with extensions to nonlinear observations and weak-observation regimes through techniques like Restricted Encoding.

Abstract

The recent emergence of diffusion models has significantly advanced the precision of learnable priors, presenting innovative avenues for addressing inverse problems. Since inverse problems inherently entail maximum a posteriori estimation, previous works have endeavored to integrate diffusion priors into the optimization frameworks. However, prevailing optimization-based inverse algorithms primarily exploit the prior information within the diffusion models while neglecting their denoising capability. To bridge this gap, this work leverages the diffusion process to reframe noisy inverse problems as a two-variable constrained optimization task by introducing an auxiliary optimization variable. By employing gradient truncation, the projection gradient descent method is efficiently utilized to solve the corresponding optimization problem. The proposed algorithm, termed ProjDiff, effectively harnesses the prior information and the denoising capability of a pre-trained diffusion model within the optimization framework. Extensive experiments on the image restoration tasks and source separation and partial generation tasks demonstrate that ProjDiff exhibits superior performance across various linear and nonlinear inverse problems, highlighting its potential for practical applications. Code is available at https://github.com/weigerzan/ProjDiff/.
Paper Structure (38 sections, 4 theorems, 67 equations, 11 figures, 21 tables, 4 algorithms)

This paper contains 38 sections, 4 theorems, 67 equations, 11 figures, 21 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Backgrounds
  3. Denoising diffusion models
  4. Diffusion models as data prior
  5. Equivalent noisy samples
  6. Method
  7. ProjDiff
  8. Noise-free observations: a special case
  9. Extension to nonlinear observation
  10. Time steps, initialization, and Restricted Encoding
  11. Experiments.
  12. Image restoration
  13. Source separation and partial generation
  14. Conclusion
  15. Derivation for ProjDiff in VP diffusion
  16. ...and 23 more sections

Key Result

Proposition 1

Considering the DDIM reference distribution $q_\sigma$, we have the variational lower bound of the log-prior term as where $C$ is a constant independent of $x_0$ and $x_{t_a}$, the weight $g(t), w(t)$ are functions of $\overline\alpha_t$ and the DDIM variance $\tilde{\sigma}_t$, and $\bm{\nu}=\left({x_t-\sqrt{\frac{\overline\alpha_t}{\overline\alpha_{t_a}}}x_{t_a}}\right)/{\sqrt{1-\frac{\overline

Figures (11)

  • Figure 1: Framework of ProjDiff. We introduce an auxiliary variable $x_{t_a}$ and transform the inverse problem into a two-variable constrained optimization problem which can be solved using the projection gradient method.
  • Figure 2: Linear restoration on CelebA ($\sigma=0.05$). Baseline means $\hat{x}_0 = \mathbf{A}^\dagger y$.
  • Figure 3: Nonlinear restoration on FFHQ (noise-free).
  • Figure 4: Noise-free super-resolution results on ImageNet. The red lines show the variation of PSNR v.s. FID and LPIPS of ProjDiff algorithm.
  • Figure 5: Samples of the partial generation results. The results from ProjDiff (left) have larger amplitude, while results from RED-diff (middle) and MSDM (right) have more silent periods.
  • ...and 6 more figures

Theorems & Definitions (8)

  • Proposition 1
  • Proposition 2
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • proof
  • proof