Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HAIR: Hypernetworks-based All-in-One Image Restoration

Jin Cao, Yi Cao, Li Pang, Deyu Meng, Xiangyong Cao

TL;DR

This work proposes HAIR, a Hypernetworks-based All-in-One Image Restoration plug-and-play method that generates parameters based on the input image and thus makes the model to adapt to specific degradation dynamically, and proposes Res-HAIR, which integrates HAIR into the well-known Restormer.

Abstract

Image restoration aims to recover a high-quality clean image from its degraded version. Recent progress in image restoration has demonstrated the effectiveness of All-in-One image restoration models in addressing various unknown degradations simultaneously. However, these existing methods typically utilize the same parameters to tackle images with different types of degradation, forcing the model to balance the performance between different tasks and limiting its performance on each task. To alleviate this issue, we propose HAIR, a Hypernetworks-based All-in-One Image Restoration plug-and-play method that generates parameters based on the input image and thus makes the model to adapt to specific degradation dynamically. Specifically, HAIR consists of two main components, i.e., Classifier and Hyper Selecting Net (HSN). The Classifier is a simple image classification network used to generate a Global Information Vector (GIV) that contains the degradation information of the input image, and the HSN is a simple fully-connected neural network that receives the GIV and outputs parameters for the corresponding modules. Extensive experiments demonstrate that HAIR can significantly improve the performance of existing image restoration models in a plug-and-play manner, both in single-task and All-in-One settings. Notably, our proposed model Res-HAIR, which integrates HAIR into the well-known Restormer, can obtain superior or comparable performance compared with current state-of-the-art methods. Moreover, we theoretically demonstrate that to achieve a given small enough error, our proposed HAIR requires fewer parameters in contrast to mainstream embedding-based All-in-One methods. The code is available at https://github.com/toummHus/HAIR.

HAIR: Hypernetworks-based All-in-One Image Restoration

TL;DR

This work proposes HAIR, a Hypernetworks-based All-in-One Image Restoration plug-and-play method that generates parameters based on the input image and thus makes the model to adapt to specific degradation dynamically, and proposes Res-HAIR, which integrates HAIR into the well-known Restormer.

Abstract

Image restoration aims to recover a high-quality clean image from its degraded version. Recent progress in image restoration has demonstrated the effectiveness of All-in-One image restoration models in addressing various unknown degradations simultaneously. However, these existing methods typically utilize the same parameters to tackle images with different types of degradation, forcing the model to balance the performance between different tasks and limiting its performance on each task. To alleviate this issue, we propose HAIR, a Hypernetworks-based All-in-One Image Restoration plug-and-play method that generates parameters based on the input image and thus makes the model to adapt to specific degradation dynamically. Specifically, HAIR consists of two main components, i.e., Classifier and Hyper Selecting Net (HSN). The Classifier is a simple image classification network used to generate a Global Information Vector (GIV) that contains the degradation information of the input image, and the HSN is a simple fully-connected neural network that receives the GIV and outputs parameters for the corresponding modules. Extensive experiments demonstrate that HAIR can significantly improve the performance of existing image restoration models in a plug-and-play manner, both in single-task and All-in-One settings. Notably, our proposed model Res-HAIR, which integrates HAIR into the well-known Restormer, can obtain superior or comparable performance compared with current state-of-the-art methods. Moreover, we theoretically demonstrate that to achieve a given small enough error, our proposed HAIR requires fewer parameters in contrast to mainstream embedding-based All-in-One methods. The code is available at https://github.com/toummHus/HAIR.
Paper Structure (27 sections, 6 theorems, 10 equations, 8 figures, 11 tables)

This paper contains 27 sections, 6 theorems, 10 equations, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. All-in-One Image Restoration
  4. Data-conditioned Hypernetworks
  5. Method
  6. Proposed HAIR Module
  7. Degradation-Aware Classifier (DAC)
  8. Hyper Selecting Net (HSN)
  9. Proposed Res-HAIR Model
  10. Theoretical Analysis
  11. Hypernetworks from a Selection Perspective
  12. A Brief Analysis of Model Complexity
  13. Experiments
  14. Experimenting Preparation
  15. Results
  16. ...and 12 more sections

Key Result

Proposition 1

(Convolution operations exhibit the distributive law over addition) Let $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ be the input feature, and let $w_i, \ i = 1, 2, \cdots, n$ represent the convolution kernels. The law is mathematically expressed as: where '*' denotes the standard 2-dimensional convolution.

Figures (8)

  • Figure 1: Comparisions between our method and previous methods in the inference stage. (a) Previous All-in-One image restoration methods. These methods utilize a single model with fixed parameters to tackle different degradations. (b) Our proposed HAIR. Given a certain degraded image, we use Hypernetworks to generate the dynamic parameters for the decoder, and finally obtain the restored image. Note that "dynamic" and "fixed" in this paper are specially for the main networks, as discussed in \ref{['app:fix_or_not']}.
  • Figure 2: Comparison of tSNE plots for the degradation embeddings between previous methods and our HAIR (i.e. the GIVs). Each distinct color represents a unique degradation type. As shown in (c), our HAIR excels not only in recognizing various degradation types, such as noise, rain, and haze, but also in distinguishing between the same type of degradation at varying intensities, e.g. noise with different standard deviations. Even when confronted with composite degradations not encountered during training, HAIR can also accurately discriminate them, i.e. the GIVs for these composite cases located midway between the GIVs of their constituent degradations, as illustrated in (d).
  • Figure 3: The overall framework of our proposed Res-HAIR. Res-HAIR is built by integrating our HAIR into the popular Restormer restormer. HAIR contains two modules, i.e. Degradation-Aware Classifier (DAC) and Hyper Selecting Net (HSN). The DAC is used to yield a Global Information Vector (GIV) $\mathbf{V_g}$ from the high-level feature $\mathbf{F_g}$ containing the degradation information of the input image. HSN dynamically generates Transformer Block weights based on GIV, tailoring restoration to each image's unique degradation.
  • Figure 4: Visual comparison on various degradation settings. Zoom for better visual effects.
  • Figure 5: Impact of the existence of DAC and HSN. Results are from single deraining task on Rain100L.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Proposition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5