LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration

TL;DR

LoRA-IR is a flexible framework that dynamically leverages compact low-rank experts to facilitate efficient all-in-one image restoration and significantly enhances the model's adaptability to diverse and unknown degradations in complex real-world scenarios.

Abstract

Prompt-based all-in-one image restoration (IR) frameworks have achieved remarkable performance by incorporating degradation-specific information into prompt modules. Nevertheless, handling the complex and diverse degradations encountered in real-world scenarios remains a significant challenge. To tackle this, we propose LoRA-IR, a flexible framework that dynamically leverages compact low-rank experts to facilitate efficient all-in-one image restoration. Specifically, LoRA-IR consists of two training stages: degradation-guided pre-training and parameter-efficient fine-tuning. In the pre-training stage, we enhance the pre-trained CLIP model by introducing a simple mechanism that scales it to higher resolutions, allowing us to extract robust degradation representations that adaptively guide the IR network. In the fine-tuning stage, we refine the pre-trained IR network through low-rank adaptation (LoRA). Built upon a Mixture-of-Experts (MoE) architecture, LoRA-IR dynamically integrates multiple low-rank restoration experts through a degradation-guided router. This dynamic integration mechanism significantly enhances our model's adaptability to diverse and unknown degradations in complex real-world scenarios. Extensive experiments demonstrate that LoRA-IR achieves SOTA performance across 14 IR tasks and 29 benchmarks, while maintaining computational efficiency. Code and pre-trained models will be available at: https://github.com/shallowdream204/LoRA-IR.

Figures (6)

• Figure 1: PSNR comparison with state-of-the-art all-in-one methods across 8 image restoration tasks (Tab. \ref{['tab:de_5']} and Tab. \ref{['tab:mixed']}).
• Figure 2: Conceptual comparison of all-in-one frameworks. (a) Multi-Encoder Structures: Use multiple encoders to extract features, but redundancy reduces model efficiency. (b) Prompt-Based Methods: Employ lightweight prompts for degradation-specific features, improving efficiency. However, static network structures limit their ability to handle unknown complex degradations. (c) Our Proposed Framework: Self-adaptively and sparsely combines low-rank restoration experts. This design preserves model efficiency while enabling self-adaptation to various degradation types, thereby enhancing its real-world performance.
• Figure 3: Visualization of images output by the CLIP processor (top row from GoPro gopro and bottom row from LIVE1 sheikh2005live), which reveals significant loss of degradation information after processing. Please zoom in for a better view.
• Figure 4: Overall of the proposed LoRA-IR, which includes (a) Degradation-guided router (DG-Router), (b) Pre-training image restoration network with robust degradation embedding, (c) Fine-tuning image restoration network with low-rank restoration experts, and (d) Degradation-guided adaptative modulator (DAM).
• Figure 5: [Setting IV] Visual results on HIDE hide for training-seen tasks generalization evaluation (top row) and TOLED udc for training-unseen tasks generalization evaluation (bottom row). Zoom in for a better view.