Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Diffusion Posterior Sampling for Noisy Inverse Problems

Ji Li, Chao Wang

TL;DR

The paper tackles noisy inverse problems by leveraging pretrained diffusion models as priors and proposing a MAP-based surrogate to estimate the conditional mean $\mathbb{E}[\mathbf{x}_0|\mathbf{x}_t,\mathbf{y}]$ without backpropagation. It introduces a DDIM-like, parameterized sampler with measurement guidance that uses the MAP surrogate to drive conditional Sampling, enabling SVD-free and backprop-free implementation. The approach unifies and differentiates from existing methods such as $\Pi$GDM, DMPS, and DDRM, while extending to nonlinear cases like JPEG decompression; experiments show competitive restoration quality and notable runtime advantages. This yields a practical, flexible framework for broad inverse problems, preserving diffusion priors while reducing computational and memory costs. The work thus enhances applicability of diffusion-based priors for real-world linear and nonlinear inverse problems with improved efficiency and generality.

Abstract

The pretrained diffusion model as a strong prior has been leveraged to address inverse problems in a zero-shot manner without task-specific retraining. Different from the unconditional generation, the measurement-guided generation requires estimating the expectation of clean image given the current image and the measurement. With the theoretical expectation expression, the crucial task of solving inverse problems is to estimate the noisy likelihood function at the intermediate image sample. Using the Tweedie's formula and the known noise model, the existing diffusion posterior sampling methods perform gradient descent step with backpropagation through the pretrained diffusion model. To alleviate the costly computation and intensive memory consumption of the backpropagation, we propose an alternative maximum-a-posteriori (MAP)-based surrogate estimator to the expectation. With this approach and further density approximation, the MAP estimator for linear inverse problem is the solution to a traditional regularized optimization, of which the loss comprises of data fidelity term and the diffusion model related prior term. Integrating the MAP estimator into a general denoising diffusion implicit model (DDIM)-like sampler, we achieve the general solving framework for inverse problems. Our approach highly resembles the existing $Π$GDM without the manifold projection operation of the gradient descent direction. The developed method is also extended to nonlinear JPEG decompression. The performance of the proposed posterior sampling is validated across a series of inverse problems, where both VP and VE SDE-based pretrained diffusion models are taken into consideration.

Efficient Diffusion Posterior Sampling for Noisy Inverse Problems

TL;DR

The paper tackles noisy inverse problems by leveraging pretrained diffusion models as priors and proposing a MAP-based surrogate to estimate the conditional mean without backpropagation. It introduces a DDIM-like, parameterized sampler with measurement guidance that uses the MAP surrogate to drive conditional Sampling, enabling SVD-free and backprop-free implementation. The approach unifies and differentiates from existing methods such as GDM, DMPS, and DDRM, while extending to nonlinear cases like JPEG decompression; experiments show competitive restoration quality and notable runtime advantages. This yields a practical, flexible framework for broad inverse problems, preserving diffusion priors while reducing computational and memory costs. The work thus enhances applicability of diffusion-based priors for real-world linear and nonlinear inverse problems with improved efficiency and generality.

Abstract

The pretrained diffusion model as a strong prior has been leveraged to address inverse problems in a zero-shot manner without task-specific retraining. Different from the unconditional generation, the measurement-guided generation requires estimating the expectation of clean image given the current image and the measurement. With the theoretical expectation expression, the crucial task of solving inverse problems is to estimate the noisy likelihood function at the intermediate image sample. Using the Tweedie's formula and the known noise model, the existing diffusion posterior sampling methods perform gradient descent step with backpropagation through the pretrained diffusion model. To alleviate the costly computation and intensive memory consumption of the backpropagation, we propose an alternative maximum-a-posteriori (MAP)-based surrogate estimator to the expectation. With this approach and further density approximation, the MAP estimator for linear inverse problem is the solution to a traditional regularized optimization, of which the loss comprises of data fidelity term and the diffusion model related prior term. Integrating the MAP estimator into a general denoising diffusion implicit model (DDIM)-like sampler, we achieve the general solving framework for inverse problems. Our approach highly resembles the existing GDM without the manifold projection operation of the gradient descent direction. The developed method is also extended to nonlinear JPEG decompression. The performance of the proposed posterior sampling is validated across a series of inverse problems, where both VP and VE SDE-based pretrained diffusion models are taken into consideration.

Paper Structure

This paper contains 29 sections, 5 theorems, 47 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Traditional hand-crafted prior for inverse problems
  4. Supervise learning for inverse problems
  5. GAN inversion and conditional normalizing flow
  6. Diffusion models for inverse problems
  7. Direct diffusion models for inverse problems
  8. Plug-and-play methods for inverse problems
  9. Background
  10. SDE modeling for diffusion models
  11. Inverse problems using diffusion models
  12. Tweedie's formula for denoising
  13. Diffusion posterior sampling in discretization case
  14. Existing approximations of the likelihood and their issues
  15. Methodology
  16. ...and 14 more sections

Key Result

Lemma 3.2

\newlabellem:10 Given the Gaussian conditional density $p(\bm{x}_t|\bm{x}_0)\sim\mathcal{N}(\bm{x}_t|\alpha_t\bm{x}_0,\sigma_t^2\bm{I})$, the mean and variance of $p(\bm{x}_0|\bm{x}_t)$ are given by

Figures (7)

  • Figure 1: Visualization of the results from our method with different $\xi$ for Gaussian noisy linear problems on the FFHQ $256\times 256$. Gaussian noise ($\sigma_y=0.05$) is added to the measurement. \newlabelfig:ffhq_gauss_x0
  • Figure 2: Visualization of the results from our method with different $\xi$ for Gaussian noisy linear problems on the ImageNet $256\times 256$. Gaussian noise ($\sigma_y=0.05$) is added to the measurement. \newlabelfig:img_gauss_x0
  • Figure 3: Visualization of the results from different methods for Gaussian noisy linear problems on the FFHQ $256\times 256$. Gaussian noise ($\sigma_y=0.05$) is added to the measurement. \newlabelfig:ffhq_gauss0
  • Figure 4: Visualization of the results from different methods for Gaussian noisy linear problems on the ImageNet $256\times 256$. Gaussian noise ($\sigma_y=0.05$) is added to the measurement. \newlabelfig:img_gauss0
  • Figure 5: Results for noisy motion deblurring. \newlabelfig:motion_deblur0
  • ...and 2 more figures

Theorems & Definitions (9)

  • Remark 3.1: ODE modeling for diffusion model
  • Lemma 3.2: Mean and variance of $p(\bm{x}_0|\bm{x}_t)$ boys2023tweedie
  • Theorem 4.1: peng2024imp \newlabelthm:10
  • Lemma 4.2
  • Proof 1
  • Theorem 4.3
  • Lemma 4.4: Upper bound of TV distance of two multivariate Gaussians arbas2023polynomial
  • Proof 2: Proof of Theorem \ref{['thm:2']}
  • Remark 5.1: Why DMPS fails for inpainting?