Degradation-Aware Feature Perturbation for All-in-One Image Restoration

TL;DR

The paper tackles the problem of consistent restoration across multiple degradations with a single model, addressing task interference that arises when shared parameters must handle diverse degradation features. It introduces Degradation-aware Feature Perturbation (DFPIR), which perturbs feature spaces using a Degradation-Guided Perturbation Block (DGPB) composed of a Degradation-Guided Channel Perturbation Module (DGCPM) and a Channel-Adapted Attention Perturbation Module (CAAPM), guided by degradation prompts encoded via CLIP. This design, including channel shuffling in a high-dimensional space and selective attention masking, aligns the feature space with a unified parameter space and reduces cross-task interference, yielding state-of-the-art results on all-in-one restoration tasks across three and five degradation types, with substantial PSNR gains over prior methods. The approach demonstrates strong empirical gains, clear visual improvements, and interpretable perturbation mechanisms, suggesting practical impact for robust, multi-degradation image restoration with a single model.

Abstract

All-in-one image restoration aims to recover clear images from various degradation types and levels with a unified model. Nonetheless, the significant variations among degradation types present challenges for training a universal model, often resulting in task interference, where the gradient update directions of different tasks may diverge due to shared parameters. To address this issue, motivated by the routing strategy, we propose DFPIR, a novel all-in-one image restorer that introduces Degradation-aware Feature Perturbations(DFP) to adjust the feature space to align with the unified parameter space. In this paper, the feature perturbations primarily include channel-wise perturbations and attention-wise perturbations. Specifically, channel-wise perturbations are implemented by shuffling the channels in high-dimensional space guided by degradation types, while attention-wise perturbations are achieved through selective masking in the attention space. To achieve these goals, we propose a Degradation-Guided Perturbation Block (DGPB) to implement these two functions, positioned between the encoding and decoding stages of the encoder-decoder architecture. Extensive experimental results demonstrate that DFPIR achieves state-of-the-art performance on several all-in-one image restoration tasks including image denoising, image dehazing, image deraining, motion deblurring, and low-light image enhancement. Our codes are available at https://github.com/TxpHome/DFPIR.

Figures (7)

• Figure 1: This figure demonstrates the channel-wise perturbation method of DFPIR, where channel shuffle assigns unique channel orders for different degradation types.
• Figure 2: This figure showcases our attention-wise perturbation, where DFPIR applies attention selection to perturb image features, discarding a portion of attention for each degradation type.
• Figure 3: The figure provides t-SNE plots of the intermediate features from DFPIR (our method) and PromptIR on the test datasets (CBSD68, Rain100L, and SOTS) under the three-task setting. In our model, the features for each task exhibit tighter clustering, highlighting the effectiveness of our degradation-aware feature perturbation strategy in enhancing restoration performance.
• Figure 4: Overview of the DFPIR framework. We employ Restormer IM_zamir2022restormer, an encoder-decoder network with transformer blocks in the encoding and decoding stages, as our backbone. The primary component of the framework, the Degradation-Guided Perturbation Block (DGPB), consists of two submodules, i.e., Degradation-Guided Channel Perturbation Module (DGCPM) and Channel-Adapted Attention Perturbation Module (CAAPM). The DGCPM module introduces conventional dimensional perturbations to image features in the form of channel shuffling, guided by degradation-type prompts. The CAAPM module applies attention perturbation to the channel-shuffled features through a top-K masking strategy.
• Figure 5: Visual comparison of DFPIR with state-of-the-art methods on challenging cases for the All-in-One setting considering three degradations. Zoom in for better view.