Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts

TL;DR

The proposed AdaptIR is a Mixture-of-Experts (MoE) with orthogonal multi-branch design to capture local spatial, global spatial, and channel representation bases, followed by adaptive base combination to obtain heterogeneous representation for different degradations.

Abstract

Designing single-task image restoration models for specific degradation has seen great success in recent years. To achieve generalized image restoration, all-in-one methods have recently been proposed and shown potential for multiple restoration tasks using one single model. Despite the promising results, the existing all-in-one paradigm still suffers from high computational costs as well as limited generalization on unseen degradations. In this work, we introduce an alternative solution to improve the generalization of image restoration models. Drawing inspiration from recent advancements in Parameter Efficient Transfer Learning (PETL), we aim to tune only a small number of parameters to adapt pre-trained restoration models to various tasks. However, current PETL methods fail to generalize across varied restoration tasks due to their homogeneous representation nature. To this end, we propose AdaptIR, a Mixture-of-Experts (MoE) with orthogonal multi-branch design to capture local spatial, global spatial, and channel representation bases, followed by adaptive base combination to obtain heterogeneous representation for different degradations. Extensive experiments demonstrate that our AdaptIR achieves stable performance on single-degradation tasks, and excels in hybrid-degradation tasks, with fine-tuning only 0.6% parameters for 8 hours.

Figures (17)

• Figure 1: (a)&(b) We find directly applying the current PETL methods to image restoration leads to unstable performance on single degradation. (c) The current PETL method suffers sub-optimal results on hybrid degradation which requires heterogeneous representation. (d)&(e) We use Fourier analysis to visualize Adapter and our AdaptIR and find that Adapter exhibits homogeneous frequency representations even when faced with different degradations, while our AdaptIR can adaptively learn degradation-specific heterogeneous representations. We provide more evidence in \ref{['sec:more-evidence']}.
• Figure 2: An illustration of the proposed AdaptIR. Our AdaptIR is placed parallel to the frozen MLP in one transformer layer and thus can be seamlessly inserted into various transformer-based pre-trained restoration models.
• Figure 3: Visual comparison on hybrid degradation with LR4&Noise30. We provide more visualization in \ref{['sec:single_task']}.
• Figure 4: Fouriur analysis on outputs from LIM and FAM.
• Figure 4: Comparison on generalization ability with more pretrained base model.