Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Low-Pass Filtering Behavior in Image Super-Resolution

Haoyu Deng, Zijing Xu, Yule Duan, Xiao Wu, Wenjie Shu, Liang-Jian Deng

TL;DR

This paper reports an intriguing phenomenon, referred to as `the sinc phenomenon,' and proposes a method named Hybrid Response Analysis (HyRA) to analyze the behavior of neural networks in ISR tasks, which decomposes a neural network into a parallel connection of a linear system and a non-linear system.

Abstract

Deep neural networks for image super-resolution (ISR) have shown significant advantages over traditional approaches like the interpolation. However, they are often criticized as 'black boxes' compared to traditional approaches with solid mathematical foundations. In this paper, we attempt to interpret the behavior of deep neural networks in ISR using theories from the field of signal processing. First, we report an intriguing phenomenon, referred to as `the sinc phenomenon.' It occurs when an impulse input is fed to a neural network. Then, building on this observation, we propose a method named Hybrid Response Analysis (HyRA) to analyze the behavior of neural networks in ISR tasks. Specifically, HyRA decomposes a neural network into a parallel connection of a linear system and a non-linear system and demonstrates that the linear system functions as a low-pass filter while the non-linear system injects high-frequency information. Finally, to quantify the injected high-frequency information, we introduce a metric for image-to-image tasks called Frequency Spectrum Distribution Similarity (FSDS). FSDS reflects the distribution similarity of different frequency components and can capture nuances that traditional metrics may overlook. Code, videos and raw experimental results for this paper can be found in: https://github.com/RisingEntropy/LPFInISR.

Exploring the Low-Pass Filtering Behavior in Image Super-Resolution

TL;DR

This paper reports an intriguing phenomenon, referred to as `the sinc phenomenon,' and proposes a method named Hybrid Response Analysis (HyRA) to analyze the behavior of neural networks in ISR tasks, which decomposes a neural network into a parallel connection of a linear system and a non-linear system.

Abstract

Deep neural networks for image super-resolution (ISR) have shown significant advantages over traditional approaches like the interpolation. However, they are often criticized as 'black boxes' compared to traditional approaches with solid mathematical foundations. In this paper, we attempt to interpret the behavior of deep neural networks in ISR using theories from the field of signal processing. First, we report an intriguing phenomenon, referred to as `the sinc phenomenon.' It occurs when an impulse input is fed to a neural network. Then, building on this observation, we propose a method named Hybrid Response Analysis (HyRA) to analyze the behavior of neural networks in ISR tasks. Specifically, HyRA decomposes a neural network into a parallel connection of a linear system and a non-linear system and demonstrates that the linear system functions as a low-pass filter while the non-linear system injects high-frequency information. Finally, to quantify the injected high-frequency information, we introduce a metric for image-to-image tasks called Frequency Spectrum Distribution Similarity (FSDS). FSDS reflects the distribution similarity of different frequency components and can capture nuances that traditional metrics may overlook. Code, videos and raw experimental results for this paper can be found in: https://github.com/RisingEntropy/LPFInISR.
Paper Structure (33 sections, 1 theorem, 22 equations, 19 figures, 6 tables)

This paper contains 33 sections, 1 theorem, 22 equations, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Super Resolution Using Neural Networks
  4. Explaining the Behavior of Neural Networks
  5. Preliminaries
  6. Method
  7. Hybrid Response Analysis (HyRA)
  8. $H(I)$ is a low-pass filter
  9. $G(I)$ Injects high-frequency information
  10. Frequency Spectrum Distribution Similarity (FSDS)
  11. Motivation and Method
  12. The merits of FSDS
  13. Experiments
  14. Experiment on Low-pass Filtering Super-resolution Performance
  15. Experiment on Impulse Response
  16. ...and 18 more sections

Key Result

Lemma 4.1

A neural network $N(I)$ can be expressed as a combination of a linear system $H(I)$ and a non-linear system with an impulse response of zero, i.e., $N(I) = H(I) + G(I)$, where $G(\delta) = 0$. Here, $\delta$ represents the Dirac delta function.

Figures (19)

  • Figure 1: $I$ is an image in which only the central pixel is 1 and the other pixels are 0. What would the result look like if image I is super-resolved using a neural network, A, B, C, or D? Surprisingly, the answer is A. We name this phenomenon as the sinc phenomenon. In this paper, we give a possible explanation for this phenomenon.
  • Figure 2: Various interpolation kernels for ISR. They can all be seen as an approximation of sinc function.
  • Figure 3: Conceptual diagram of HyRA's core idea.
  • Figure 4: Top row: a super-resolved image by edsr can be viewed as the summation of a linear response obtained by convolving impulse response with the input and the non-linear response gained by subtracting linear-part from the ISR result. Second row: the corresponding frequency spectrum amplitude of the top row. Third row: the corresponding frequency spectrum phase of the top row. The phase compensation indicates that the non-linear part is compensating distortion.
  • Figure 5: An illustration of how the passband width of a low-pass filter affects its ISR results. A too wide passband or a too narrow passband can result in a decline in performance.
  • ...and 14 more figures

Theorems & Definitions (3)

  • proof
  • Lemma 4.1
  • proof