Deep LoRA-Unfolding Networks for Image Restoration

TL;DR

LoRun introduces a novel paradigm where a single pretrained base denoiser is shared across all stages, while lightweight, stage-specific LoRA adapters are injected into the PMMs to dynamically modulate denoising behavior according to the noise level at each unfolding step, enabling precise control over denoising intensity without duplicating full network parameters.

Abstract

Deep unfolding networks (DUNs), combining conventional iterative optimization algorithms and deep neural networks into a multi-stage framework, have achieved remarkable accomplishments in Image Restoration (IR), such as spectral imaging reconstruction, compressive sensing and super-resolution.It unfolds the iterative optimization steps into a stack of sequentially linked blocks.Each block consists of a Gradient Descent Module (GDM) and a Proximal Mapping Module (PMM) which is equivalent to a denoiser from a Bayesian perspective, operating on Gaussian noise with a known level.However, existing DUNs suffer from two critical limitations: (i) their PMMs share identical architectures and denoising objectives across stages, ignoring the need for stage-specific adaptation to varying noise levels; and (ii) their chain of structurally repetitive blocks results in severe parameter redundancy and high memory consumption, hindering deployment in large-scale or resource-constrained scenarios.To address these challenges, we introduce generalized Deep Low-rank Adaptation (LoRA) Unfolding Networks for image restoration, named LoRun, harmonizing denoising objectives and adapting different denoising levels between stages with compressed memory usage for more efficient DUN.LoRun introduces a novel paradigm where a single pretrained base denoiser is shared across all stages, while lightweight, stage-specific LoRA adapters are injected into the PMMs to dynamically modulate denoising behavior according to the noise level at each unfolding step.This design decouples the core restoration capability from task-specific adaptation, enabling precise control over denoising intensity without duplicating full network parameters and achieving up to $N$ times parameter reduction for an $N$-stage DUN with on-par or better performance.Extensive experiments conducted on three IR tasks validate the efficiency of our method.

Figures (17)

• Figure 1: PSNR-Params.-FLOPs comparison of our LoRun and SOTAs in CASSI task. Our LoRun reduces the parameters of traditional DUN methods by nearly 70% while maintaining comparable performance.
• Figure 2: Structure comparison between the previous DUNs and our hierarchical deep LoRA-unfolding networks. Previous DUNs adopt independent parameters to all stages while our LoRun first leverages pre-trained denoiser for each stage then is trained with different LoRA layers for different denoising abilities.
• Figure 3: Representation for LoRA adopted in our LoRun framework.
• Figure 4: Illustration of the proposed LoRun method. (a) The gradient descent module. (a-1) The gradient descent process based on the PGD algorithm. (a-2) The gradient descent process based on the HQS algorithm. (b) The proximal mapping module (denoiser). (c) Framework of the proposed deep LoRA-Unfolding network. Our LoRun first adopts the same most frozen-shared backbone denoiser among all stages, followed by fine-tuning the denoising level and direction of each stage with small learnable-separate low-rank modules. This strategy mostly reduces the parameters and accelerates convergence.
• Figure 5: CS results under ratio $\beta=25\%$ of different methods and datasets, from top to bottom General100, Set11 and Set14.