Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion Posterior Proximal Sampling for Image Restoration

Hongjie Wu, Linchao He, Mingqin Zhang, Dongdong Chen, Kunming Luo, Mengting Luo, Ji-Zhe Zhou, Hu Chen, Jiancheng Lv

TL;DR

This work tackles image restoration with diffusion models by addressing randomness-induced instability. It introduces Diffusion Posterior Proximal Sampling (DPPS), which at each reverse step draws multiple candidates and selects the one most consistent with the observed measurement, while adapting the number of candidates over time and starting from a measurement-informed initialization. The method achieves faster convergence, improved perceptual quality (LPIPS/FID) on diverse tasks such as super-resolution, deblurring, and inpainting, with only modest computational overhead. The results demonstrate that aligning diffusion sampling with measurement identity yields robust restoration under varying conditions, offering practical benefits for real-world inverse problems while leaving some distortion-metric trade-offs to be explored.

Abstract

Diffusion models have demonstrated remarkable efficacy in generating high-quality samples. Existing diffusion-based image restoration algorithms exploit pre-trained diffusion models to leverage data priors, yet they still preserve elements inherited from the unconditional generation paradigm. These strategies initiate the denoising process with pure white noise and incorporate random noise at each generative step, leading to over-smoothed results. In this paper, we present a refined paradigm for diffusion-based image restoration. Specifically, we opt for a sample consistent with the measurement identity at each generative step, exploiting the sampling selection as an avenue for output stability and enhancement. The number of candidate samples used for selection is adaptively determined based on the signal-to-noise ratio of the timestep. Additionally, we start the restoration process with an initialization combined with the measurement signal, providing supplementary information to better align the generative process. Extensive experimental results and analyses validate that our proposed method significantly enhances image restoration performance while consuming negligible additional computational resources.

Diffusion Posterior Proximal Sampling for Image Restoration

TL;DR

This work tackles image restoration with diffusion models by addressing randomness-induced instability. It introduces Diffusion Posterior Proximal Sampling (DPPS), which at each reverse step draws multiple candidates and selects the one most consistent with the observed measurement, while adapting the number of candidates over time and starting from a measurement-informed initialization. The method achieves faster convergence, improved perceptual quality (LPIPS/FID) on diverse tasks such as super-resolution, deblurring, and inpainting, with only modest computational overhead. The results demonstrate that aligning diffusion sampling with measurement identity yields robust restoration under varying conditions, offering practical benefits for real-world inverse problems while leaving some distortion-metric trade-offs to be explored.

Abstract

Diffusion models have demonstrated remarkable efficacy in generating high-quality samples. Existing diffusion-based image restoration algorithms exploit pre-trained diffusion models to leverage data priors, yet they still preserve elements inherited from the unconditional generation paradigm. These strategies initiate the denoising process with pure white noise and incorporate random noise at each generative step, leading to over-smoothed results. In this paper, we present a refined paradigm for diffusion-based image restoration. Specifically, we opt for a sample consistent with the measurement identity at each generative step, exploiting the sampling selection as an avenue for output stability and enhancement. The number of candidate samples used for selection is adaptively determined based on the signal-to-noise ratio of the timestep. Additionally, we start the restoration process with an initialization combined with the measurement signal, providing supplementary information to better align the generative process. Extensive experimental results and analyses validate that our proposed method significantly enhances image restoration performance while consuming negligible additional computational resources.
Paper Structure (42 sections, 3 theorems, 27 equations, 15 figures, 6 tables, 1 algorithm)

This paper contains 42 sections, 3 theorems, 27 equations, 15 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Denoising Diffusion Probabilistic Models
  4. Diffusion Based Solvers for Image Restoration
  5. Diffusion Posterior Proximal Sampling
  6. Random Sampling Induces Uncertainties and Error Accumulation
  7. Proximal Sampling at Each Step
  8. Adaptive Sampling Frequency
  9. Aligned Initialization
  10. Discussion
  11. Relevance to Existing Methods.
  12. Computational Efficiency.
  13. Theoretical Analysis.
  14. Related Work
  15. Diffusion Models for Image Restoration
  16. ...and 27 more sections

Key Result

proposition 1

For the random variable $\mathbf{z} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ and its objective function: The variance for $f(\mathbf{z})$ is denoted by $\text{Var}\left(f(\mathbf{z})\right)$. We have Here, $\text{Var}_\text{DPS}\left(f(\mathbf{z})\right)$ is the variance of DPS chung2023diffusion, $\text{Var}_\text{MC}\left(f(\mathbf{z})\right)$ is the variance of Monte Carlo sampling, and $\tex

Figures (15)

  • Figure 1: Examples of the super-resolution ($\times$4) task to illustrate the efficiency of our method, where $n$ denotes the number of candidate samples, and DPS also refers to the case where $n=1$.
  • Figure 2: High-level illustration of our proposed DPPS ($n$=2). Existing methods employ random sampling from the predicted distribution, our approach takes multiple samples and selecting the one with better data consistency (denoted by $\checkmark$) at each step.
  • Figure 3: Conceptual illustration of our DPPS. Our method extract multiple candidate samples from the predicted distribution and choose the one with the highest measurement consistency.
  • Figure 4: Image restoration results with $\sigma_y=0.01$. Row 1: SR ($\times$4), Row 2: 80% inpainting, Row 3: Gaussian deblurring.
  • Figure 5: Visual results on SR ($\times$4) task to demonstrate the efficacy of our proposed method and to explore the impact of $n$.
  • ...and 10 more figures

Theorems & Definitions (3)

  • proposition 1
  • proposition 2
  • proposition 3