UnfoldLDM: Deep Unfolding-based Blind Image Restoration with Latent Diffusion Priors

TL;DR

UnfoldLDM tackles blind image restoration by addressing degradation- and texture-related limitations of prior deep unfolding networks. It combines multi-granularity degradation-aware (MGDA) gradient steps with a degradation-resistant latent diffusion model (DR-LDM) and an over-smoothing correction transformer (OCFormer) to produce degradation-free yet texture-rich outputs. The method introduces intra-stage degradation-aware (ISDA) consistency and a two-phase training regime to learn robust priors that guide refinement across unfolding stages. Empirically, UnfoldLDM achieves state-of-the-art results across diverse BIR tasks and demonstrates strong generalization and compatibility with existing DUN-based models, with code to be released.

Abstract

Deep unfolding networks (DUNs) combine the interpretability of model-based methods with the learning ability of deep networks, yet remain limited for blind image restoration (BIR). Existing DUNs suffer from: (1) \textbf{Degradation-specific dependency}, as their optimization frameworks are tied to a known degradation model, making them unsuitable for BIR tasks; and (2) \textbf{Over-smoothing bias}, resulting from the direct feeding of gradient descent outputs, dominated by low-frequency content, into the proximal term, suppressing fine textures. To overcome these issues, we propose UnfoldLDM to integrate DUNs with latent diffusion model (LDM) for BIR. In each stage, UnfoldLDM employs a multi-granularity degradation-aware (MGDA) module as the gradient descent step. MGDA models BIR as an unknown degradation estimation problem and estimates both the holistic degradation matrix and its decomposed forms, enabling robust degradation removal. For the proximal step, we design a degradation-resistant LDM (DR-LDM) to extract compact degradation-invariant priors from the MGDA output. Guided by this prior, an over-smoothing correction transformer (OCFormer) explicitly recovers high-frequency components and enhances texture details. This unique combination ensures the final result is degradation-free and visually rich. Experiments show that our UnfoldLDM achieves a leading place on various BIR tasks and benefits downstream tasks. Moreover, our design is compatible with existing DUN-based methods, serving as a plug-and-play framework. Code will be released.

Figures (6)

• Figure 1: Comparison between existing proximal gradient-based DUN-based methods (e.g., DGUNet mou2022deep and DeepSN-Net deng2025deepsn) and our UnfoldLDM. UnfoldLDM better resists unknown degradation and eliminates the over-smoothing bias that existing DUNs suffer.
• Figure 2: Framework of our proposed UnfoldLDM.
• Figure 3: Details of MGDA, DR-LDM, and OCFormer at the $k^{th}$ stage.
• Figure 4: Visualization of image denoising. Our method restores a sharper letter "s" in "His" (left) and a more accurate "i" in "ion" (right).
• Figure 5: Visualization of image deblurring. Our method resulted in a clearer hairpin (left) and a more accurate lettering "A" (right).