Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overview frequency principle/spectral bias in deep learning

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo

TL;DR

An overview of the F-Principle is provided and some open problems for future research are proposed that inspire the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective.

Abstract

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.

Overview frequency principle/spectral bias in deep learning

TL;DR

An overview of the F-Principle is provided and some open problems for future research are proposed that inspire the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective.

Abstract

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.
Paper Structure (45 sections, 1 theorem, 85 equations, 14 figures)

This paper contains 45 sections, 1 theorem, 85 equations, 14 figures.

Table of Contents

  1. Introduction
  2. Motivation
  3. Frequency principle
  4. Empirical study of F-Principle
  5. Frequency principle in low-dimensional problems
  6. Frequency principle in high-dimensional problems
  7. Projection method
  8. Filtering method
  9. Theoretical study of F-Principle
  10. Ideal setting for analyzing activation function
  11. NTK setting and linear F-Principle
  12. NTK dynamics
  13. Eigen analysis for two-layer DNN with dense data distribution
  14. Linear F-Principle for two-layer neural network with arbitrary data distribution
  15. Spectral bias of fully-connected multi-layer networks
  16. ...and 30 more sections

Key Result

Theorem 1

When $m\rightarrow \infty$, the numerical method VarEqM is solving the problem where $\delta(x)$ represents the Dirac delta function.

Figures (14)

  • Figure 1: Illustration of the training process of a DNN. Training data are sampled from target function $\sin(x)+\sin(5x)$. Red, green and black curves indicates DNN output, $\sin(x)$, and $\sin(x)+\sin(5x)$ respectively.
  • Figure 2: 1d input. (a) $f(x)$. Inset : $|\hat{f}(k)|$. (b) $\Delta_{F}(k)$ of three important frequencies (indicated by black dots in the inset of (a)) against different training epochs. Reprinted from xu2019frequency.
  • Figure 3: F-Principle in 2-d datasets. Reprinted from xu2019frequency.
  • Figure 4: Projection method. (a, b) are for MNIST, (c, d) for CIFAR10. (a, c) Amplitude $|\hat{y}_{k}|$ vs. frequency. Selected frequencies are marked by black squares. (b, d) $\Delta_{F}(k)$ vs. training epochs for the selected frequencies. Reprinted from xu2019frequency.
  • Figure 5: F-Principle in real datasets. $e_{\mathrm{low}}$ and $e_{\mathrm{high}}$ indicated by color against training epoch. Reprinted from xu2019frequency.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Theorem 1