UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation

TL;DR

The paper tackles universal image restoration across diverse degradations, including mixed degradation, where prior unified models struggle due to gradient conflicts and lack of task-specific parameters. It proposes UIR-LoRA, which freezes a large pre-trained generative model and attaches multiple low-rank adapters (LoRAs) for each degradation, with a degradation-aware router that composes LoRAs for single or mixed degradations. Through extensive experiments on multiple and mixed degradations, the approach yields higher fidelity and perceptual quality and better generalization than existing unified models, with code available. By leveraging rich priors in generative models and efficient, task-specific adapters, UIR-LoRA enables scalable handling of unseen degradations in practical restoration tasks.

Abstract

Existing unified methods typically treat multi-degradation image restoration as a multi-task learning problem. Despite performing effectively compared to single degradation restoration methods, they overlook the utilization of commonalities and specificities within multi-task restoration, thereby impeding the model's performance. Inspired by the success of deep generative models and fine-tuning techniques, we proposed a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multi-degradation restoration and transfers it to specific degradation image restoration tasks using low-rank adaptation. Additionally, we introduce a LoRA composing strategy based on the degradation similarity, which adaptively combines trained LoRAs and enables our model to be applicable for mixed degradation restoration. Extensive experiments on multiple and mixed degradations demonstrate that the proposed universal image restoration method not only achieves higher fidelity and perceptual image quality but also has better generalization ability than other unified image restoration models. Our code is available at https://github.com/Justones/UIR-LoRA.

Figures (7)

• Figure 1: Motivation of our work. A pre-trained generative model serves as the shared component and minimal parameters are added to model the specificity of each degradation restoration task.
• Figure 2: Overview of UIR-LoRA. UIR-LoRA consists of two components: a degradation-aware router and a universal image restorer. The router calculates degradation similarity in the latent space of CLIP, while the restorer utilizes the similarity provided by the router to combine LoRAs and frozen base model and restore images with multiple or mixed degadations.
• Figure 3: Qualitative comparison on multiple degraded images.
• Figure 4: Qualitative comparison on mixed degraded images from LOLBlur dataset.
• Figure 5: The impact of LoRA's rank on deblurring and denoising tasks.