Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Reweighted Least Squares with Plug-and-Play Diffusion Priors for Noisy Image Restoration

Ji Li, Chao Wang

TL;DR

This work addresses robust image restoration under general, non-Gaussian noise (including impulse noise) by casting restoration as a MAP problem with an $\ell_q$ data-fidelity term derived from a generalized Gaussian scale mixture. It couples iteratively reweighted least-squares optimization with a plug-and-play diffusion-prior proximal step, using a renoising strategy to align inputs with the diffusion model's denoising regime. The approach, applicable to diffusion and flow priors, outperforms traditional TV-based, CNN denoising, and prior diffusion-based methods across multiple datasets and tasks, with a notable improvement when using smaller $q$ values. This framework offers a flexible, extensible tool for non-Gaussian noise removal in practical imaging scenarios, leveraging score-based priors without requiring task-specific retraining.

Abstract

Existing plug-and-play image restoration methods typically employ off-the-shelf Gaussian denoisers as proximal operators within classical optimization frameworks based on variable splitting. Recently, denoisers induced by generative priors have been successfully integrated into regularized optimization methods for image restoration under Gaussian noise. However, their application to non-Gaussian noise--such as impulse noise--remains largely unexplored. In this paper, we propose a plug-and-play image restoration framework based on generative diffusion priors for robust removal of general noise types, including impulse noise. Within the maximum a posteriori (MAP) estimation framework, the data fidelity term is adapted to the specific noise model. Departing from the conventional least-squares loss used for Gaussian noise, we introduce a generalized Gaussian scale mixture-based loss, which approximates a wide range of noise distributions and leads to an $\ell_q$-norm ($0<q\leq2$) fidelity term. This optimization problem is addressed using an iteratively reweighted least squares (IRLS) approach, wherein the proximal step involving the generative prior is efficiently performed via a diffusion-based denoiser. Experimental results on benchmark datasets demonstrate that the proposed method effectively removes non-Gaussian impulse noise and achieves superior restoration performance.

Integrating Reweighted Least Squares with Plug-and-Play Diffusion Priors for Noisy Image Restoration

TL;DR

This work addresses robust image restoration under general, non-Gaussian noise (including impulse noise) by casting restoration as a MAP problem with an data-fidelity term derived from a generalized Gaussian scale mixture. It couples iteratively reweighted least-squares optimization with a plug-and-play diffusion-prior proximal step, using a renoising strategy to align inputs with the diffusion model's denoising regime. The approach, applicable to diffusion and flow priors, outperforms traditional TV-based, CNN denoising, and prior diffusion-based methods across multiple datasets and tasks, with a notable improvement when using smaller values. This framework offers a flexible, extensible tool for non-Gaussian noise removal in practical imaging scenarios, leveraging score-based priors without requiring task-specific retraining.

Abstract

Existing plug-and-play image restoration methods typically employ off-the-shelf Gaussian denoisers as proximal operators within classical optimization frameworks based on variable splitting. Recently, denoisers induced by generative priors have been successfully integrated into regularized optimization methods for image restoration under Gaussian noise. However, their application to non-Gaussian noise--such as impulse noise--remains largely unexplored. In this paper, we propose a plug-and-play image restoration framework based on generative diffusion priors for robust removal of general noise types, including impulse noise. Within the maximum a posteriori (MAP) estimation framework, the data fidelity term is adapted to the specific noise model. Departing from the conventional least-squares loss used for Gaussian noise, we introduce a generalized Gaussian scale mixture-based loss, which approximates a wide range of noise distributions and leads to an -norm () fidelity term. This optimization problem is addressed using an iteratively reweighted least squares (IRLS) approach, wherein the proximal step involving the generative prior is efficiently performed via a diffusion-based denoiser. Experimental results on benchmark datasets demonstrate that the proposed method effectively removes non-Gaussian impulse noise and achieves superior restoration performance.

Paper Structure

This paper contains 18 sections, 2 theorems, 24 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Traditional Method for Image Restoration
  4. Plug-and-play Methods
  5. Plug-and-play with Generative Prior
  6. Supervised Learning for Image Restoration
  7. Methodology
  8. Regularization via Diffusion Generative Model
  9. Majorization-minimization Algorithm
  10. Iteratively Reweighted Least-squares
  11. Updated Losses for DPS Method
  12. Applicability for Other Generative Models
  13. Numerical Experiments
  14. Conclusion
  15. Acknowledgments
  16. ...and 3 more sections

Key Result

Lemma 1

For any $x, y \in \mathbb{R}$ with $y \neq 0$ and $0 < q \leq 2$, the following inequality holds:

Figures (7)

  • Figure 1: Empirical distribution of impulse noise and its fitted approximation densities (including Laplacian, Gaussian and generalized Gaussian scale mixture) using maximum likelihood estimation. (a) The clean image; (b) Noise level $s_{sp}=0.5$, the fitted parameter $\theta=0.25, \delta=0.44,q = 1.5$. (c) Noise level $s_{sp}=0.7$, the fitted parameter $\theta=0.35, \delta=0.05,q = 0.5$. (d) Noise level $s_{sp}=0.8$, the fitted parameter $\theta=0.40, \delta=0.20,q = 0.7$. For (b)-(d), the noisy image displays on the top left corner.
  • Figure 2: Visualization of different methods on two tasks with impulse noise for FFHQ. The DPS equipped with the IRLS loss produces artifact-corrupted restoration.
  • Figure 3: Visualization of different methods on four tasks with impulse noise for FFHQ and ImageNet.
  • Figure 4: Visualization of different methods on four tasks with impulse noise for AFHQ-Cat.
  • Figure 5: Visualization of restorations from different $q$ values for image denoising on AFHQ-Cat.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Lemma 1
  • proof
  • Lemma 2
  • proof