Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Regularization-Guided Equivariant Approach for Image Restoration

Yulu Bai, Jiahong Fu, Qi Xie, Deyu Meng

TL;DR

The paper tackles the challenge of leveraging symmetry priors in image restoration without sacrificing representation accuracy. It introduces EQ-Reg, a rotation-equivariant regularizer that enforces adaptive equivariance on intermediate feature maps by self-supervised comparisons between rotated inputs and rotated feature representations. The method maintains standard CNN architectures, providing a practical, theory-grounded alternative to strict equivariant designs. Extensive experiments across metal artifact reduction, rain removal, inpainting, and classification demonstrate superior performance and generalization, underscoring the approach's broad applicability and potential to inspire future equivariant network designs.

Abstract

Equivariant and invariant deep learning models have been developed to exploit intrinsic symmetries in data, demonstrating significant effectiveness in certain scenarios. However, these methods often suffer from limited representation accuracy and rely on strict symmetry assumptions that may not hold in practice. These limitations pose a significant drawback for image restoration tasks, which demands high accuracy and precise symmetry representation. To address these challenges, we propose a rotation-equivariant regularization strategy that adaptively enforces the appropriate symmetry constraints on the data while preserving the network's representational accuracy. Specifically, we introduce EQ-Reg, a regularizer designed to enhance rotation equivariance, which innovatively extends the insights of data-augmentation-based and equivariant-based methodologies. This is achieved through self-supervised learning and the spatial rotation and cyclic channel shift of feature maps deduce in the equivariant framework. Our approach firstly enables a non-strictly equivariant network suitable for image restoration, providing a simple and adaptive mechanism for adjusting equivariance based on task. Extensive experiments across three low-level tasks demonstrate the superior accuracy and generalization capability of our method, outperforming state-of-the-art approaches.

A Regularization-Guided Equivariant Approach for Image Restoration

TL;DR

The paper tackles the challenge of leveraging symmetry priors in image restoration without sacrificing representation accuracy. It introduces EQ-Reg, a rotation-equivariant regularizer that enforces adaptive equivariance on intermediate feature maps by self-supervised comparisons between rotated inputs and rotated feature representations. The method maintains standard CNN architectures, providing a practical, theory-grounded alternative to strict equivariant designs. Extensive experiments across metal artifact reduction, rain removal, inpainting, and classification demonstrate superior performance and generalization, underscoring the approach's broad applicability and potential to inspire future equivariant network designs.

Abstract

Equivariant and invariant deep learning models have been developed to exploit intrinsic symmetries in data, demonstrating significant effectiveness in certain scenarios. However, these methods often suffer from limited representation accuracy and rely on strict symmetry assumptions that may not hold in practice. These limitations pose a significant drawback for image restoration tasks, which demands high accuracy and precise symmetry representation. To address these challenges, we propose a rotation-equivariant regularization strategy that adaptively enforces the appropriate symmetry constraints on the data while preserving the network's representational accuracy. Specifically, we introduce EQ-Reg, a regularizer designed to enhance rotation equivariance, which innovatively extends the insights of data-augmentation-based and equivariant-based methodologies. This is achieved through self-supervised learning and the spatial rotation and cyclic channel shift of feature maps deduce in the equivariant framework. Our approach firstly enables a non-strictly equivariant network suitable for image restoration, providing a simple and adaptive mechanism for adjusting equivariance based on task. Extensive experiments across three low-level tasks demonstrate the superior accuracy and generalization capability of our method, outperforming state-of-the-art approaches.

Paper Structure

This paper contains 13 sections, 2 theorems, 8 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Strict Equivariant CNNs
  4. Soft Equivariant CNNs
  5. Method Framework
  6. Prior Knowledge
  7. Proposed Regularizer
  8. Experimental Results
  9. Metal Artifact Reduction in CT Image
  10. Single Image Rain Removal
  11. Image Inpainting
  12. Image classification
  13. Conclusion

Key Result

Lemma 1

A feed forward neural network $\mathcal{N}$ is equivariant to the action of a compact group G on its inputs if and only if each layer of $\mathcal{N}$ implements a generalized form of convolution derived from the following formula. where f and $g$ are two functions $G \rightarrow \mathbb{C}$, and integration is with respect to the Haar measure $\mu$.

Figures (5)

  • Figure 1: The feature map of the trained neural network for CNN+Data Aug., EQ-CNNxie2022fourier, and the proposed EQ-Reg.
  • Figure 2: Illustrations of the feature maps of different CNNs when rotating the input images for $2\pi/3$. (a) A Common CNN. (b) The rotation EQ-CNN. (c) The proposed EQ-Regularization CNN.
  • Figure 3: Performance comparison on a typical metal-corrupted CT image from the synthesized DeepLesionyan2018deeplesion. The red pixels stand for metallic implants.
  • Figure 4: The $1^{st}$ column: a typical ground truth sample in Rain100L yang2019joint dataset (upper) and its ground truth rain layer (lower). The $2^{nd}-14^{th}$ columns: derained results (upper) and extracted rain layers (lower) by all competing methods
  • Figure 5: Inpainting reconstructions on test images with Poisson noise ($\gamma$ = 0.05) and 30% mask rate. PSNR values are shown in the top right corner of the images.

Theorems & Definitions (2)

  • Lemma 1
  • Lemma 2