Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer

Dongqi Fan, Xin Zhao, Liang Chang

TL;DR

This paper conducts in-depth analyses and proposes a novel module for image restoration called ConStyle, which can be efficiently integrated into any U-Net structure network and significantly enhance performance.

Abstract

Recently, the contrastive learning paradigm has achieved remarkable success in high-level tasks such as classification, detection, and segmentation. However, contrastive learning applied in low-level tasks, like image restoration, is limited, and its effectiveness is uncertain. This raises a question: Why does the contrastive learning paradigm not yield satisfactory results in image restoration? In this paper, we conduct in-depth analyses and propose three guidelines to address the above question. In addition, inspired by style transfer and based on contrastive learning, we propose a novel module for image restoration called \textbf{ConStyle}, which can be efficiently integrated into any U-Net structure network. By leveraging the flexibility of ConStyle, we develop a \textbf{general restoration network} for image restoration. ConStyle and the general restoration network together form an image restoration framework, namely \textbf{IRConStyle}. To demonstrate the capability and compatibility of ConStyle, we replace the general restoration network with transformer-based, CNN-based, and MLP-based networks, respectively. We perform extensive experiments on various image restoration tasks, including denoising, deblurring, deraining, and dehazing. The results on 19 benchmarks demonstrate that ConStyle can be integrated with any U-Net-based network and significantly enhance performance. For instance, ConStyle NAFNet significantly outperforms the original NAFNet on SOTS outdoor (dehazing) and Rain100H (deraining) datasets, with PSNR improvements of 4.16 dB and 3.58 dB with 85% fewer parameters.

IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer

TL;DR

This paper conducts in-depth analyses and proposes a novel module for image restoration called ConStyle, which can be efficiently integrated into any U-Net structure network and significantly enhance performance.

Abstract

Recently, the contrastive learning paradigm has achieved remarkable success in high-level tasks such as classification, detection, and segmentation. However, contrastive learning applied in low-level tasks, like image restoration, is limited, and its effectiveness is uncertain. This raises a question: Why does the contrastive learning paradigm not yield satisfactory results in image restoration? In this paper, we conduct in-depth analyses and propose three guidelines to address the above question. In addition, inspired by style transfer and based on contrastive learning, we propose a novel module for image restoration called \textbf{ConStyle}, which can be efficiently integrated into any U-Net structure network. By leveraging the flexibility of ConStyle, we develop a \textbf{general restoration network} for image restoration. ConStyle and the general restoration network together form an image restoration framework, namely \textbf{IRConStyle}. To demonstrate the capability and compatibility of ConStyle, we replace the general restoration network with transformer-based, CNN-based, and MLP-based networks, respectively. We perform extensive experiments on various image restoration tasks, including denoising, deblurring, deraining, and dehazing. The results on 19 benchmarks demonstrate that ConStyle can be integrated with any U-Net-based network and significantly enhance performance. For instance, ConStyle NAFNet significantly outperforms the original NAFNet on SOTS outdoor (dehazing) and Rain100H (deraining) datasets, with PSNR improvements of 4.16 dB and 3.58 dB with 85% fewer parameters.
Paper Structure (16 sections, 7 equations, 5 figures, 7 tables)

This paper contains 16 sections, 7 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contrastive Learning
  4. Image Restoration
  5. Style Transfer
  6. Method
  7. Analysis of Contrastive Learning in IR
  8. Image Restoration Framework: IRConStyle
  9. ConStyle
  10. ConStyle with Other Network
  11. Experiments
  12. Experimental Settings
  13. Model Analyses
  14. Experiment Results
  15. Ablation Studies
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The style transfer experiment by StyleFlow 19fan2022styleflow is performed to observe whether specific degradation of the degraded image could be transferred to a clean image.
  • Figure 2: The overall architecture of IRConStyle ((a)(b)). (a) ConStyle. (b) General restoration network, where the Preprocess, Process, and Finetune modules can be replaced with any specific operator. (c) Under the influence of Con. (contrastive), content, and style loss, the latent feature keeps moving closer to clean space and away from degradation.
  • Figure 3: The comparison of CL part in different networks. (a) MoCo. (b) MoCo adopted in DASR and AirNet. (c) ConStyle. (d) The momentum encoder and encoder architecture in ConStyle. Where Con. stands for Contrastive.
  • Figure 4: The detailed structure of the original models (a)(b)(c) and the ConStyle models (d)(e)(f). DC represents the downsample and concat operation, and UC represents upsample and concat operation.
  • Figure 5: The comparison of deblurring, dehazing, denoising, and deraining between ConStyle models and original models.