RELD: Regularization by Latent Diffusion Models for Image Restoration

TL;DR

RELD addresses ill-posed image restoration problems (denoising, deblurring, super-resolution) by integrating a Latent Diffusion Model trained for denoising into a Half-Quadratic Splitting variational framework. The method operates in the latent space via a latent diffusion prior, solving two subproblems per HQS iteration: a closed-form $\mathbf{t}$-update and a gradient-based $\mathbf{v}$-update, with a warm-start embedding of the observed data $\mathbf{b}$ through a pre-trained encoder. Empirical results on Set5 and SIDD-derived patches show RELD achieving competitive PSNR while delivering superior perceptual quality (NIQE, PIQE, LPIPS) compared with state-of-the-art diffusion-based and PnP/RED methods, while reducing computational burden thanks to latent-space optimization. The work demonstrates the practicality of latent priors for image restoration and points to future theoretical grounding and analysis of the single-gradient-step approximation.</br>All mathematical expressions are presented in $...$ notation, reflecting the paper’s formal HQS framework and latent-diffusion mechanics.

Abstract

In recent years, Diffusion Models have become the new state-of-the-art in deep generative modeling, ending the long-time dominance of Generative Adversarial Networks. Inspired by the Regularization by Denoising principle, we introduce an approach that integrates a Latent Diffusion Model, trained for the denoising task, into a variational framework using Half-Quadratic Splitting, exploiting its regularization properties. This approach, under appropriate conditions that can be easily met in various imaging applications, allows for reduced computational cost while achieving high-quality results. The proposed strategy, called Regularization by Latent Denoising (RELD), is then tested on a dataset of natural images, for image denoising, deblurring, and super-resolution tasks. The numerical experiments show that RELD is competitive with other state-of-the-art methods, particularly achieving remarkable results when evaluated using perceptual quality metrics.

Figures (5)

• Figure 1: Denoising LDM architecture training pipeline.
• Figure 2: Reconstruction results of RELD with different combinations of the hyperparameters $\gamma$ and $\mu_{0}$.
• Figure 3: Reconstruction results of RELD with different diffusion step $p$. The PSNR values are reported on top of each image.
• Figure 4: Qualitative results of image deblurring. The corrupted image is obtained setting $\sigma_{\mathbf{A}}=0.7$ and $\sigma_{\bm{\eta}}=35$.
• Figure 5: Qualitative results of $4\times$ SR. The corrupted image is obtained by setting $d=4$, $\sigma_{\mathbf{A}}=1$ and $\sigma_{\bm{\eta}}=5$.