Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On understanding and overcoming spectral biases of deep neural network learning methods for solving PDEs

Zhi-Qin John Xu, Lulu Zhang, Wei Cai

TL;DR

This work addresses the spectral bias of deep neural networks in solving PDEs, where learning favors low-frequency components and struggles with wide-band solutions. It surveys frequency-domain strategies—phase-based frequency shifting, frequency scaling via multiscale networks and Fourier features, activation optimization, random features, and hybrid/neural-operator approaches—and analyzes their mechanisms through NTK theory, diffusion models, and phase dynamics. The authors connect these modern DNN strategies to classical multiscale ideas (multigrid, wavelets) and present concrete architectures (PhaseDNN, MscaleDNN, RAFs, RFMs, MgNet) that reduce bias and promote frequency-uniform learning. The work highlights practical implications for high-dimensional, complex-geometry PDEs and outlines open problems in theoretical analysis, adaptive scaling, and robust solver design. Overall, it provides a roadmap for building DNN-based PDE solvers with uniform frequency representation, bridging deep learning with established multiscale numerical methods.

Abstract

In this review, we survey the latest approaches and techniques developed to overcome the spectral bias towards low frequency of deep neural network learning methods in learning multiple-frequency solutions of partial differential equations. Open problems and future research directions are also discussed.

On understanding and overcoming spectral biases of deep neural network learning methods for solving PDEs

TL;DR

This work addresses the spectral bias of deep neural networks in solving PDEs, where learning favors low-frequency components and struggles with wide-band solutions. It surveys frequency-domain strategies—phase-based frequency shifting, frequency scaling via multiscale networks and Fourier features, activation optimization, random features, and hybrid/neural-operator approaches—and analyzes their mechanisms through NTK theory, diffusion models, and phase dynamics. The authors connect these modern DNN strategies to classical multiscale ideas (multigrid, wavelets) and present concrete architectures (PhaseDNN, MscaleDNN, RAFs, RFMs, MgNet) that reduce bias and promote frequency-uniform learning. The work highlights practical implications for high-dimensional, complex-geometry PDEs and outlines open problems in theoretical analysis, adaptive scaling, and robust solver design. Overall, it provides a roadmap for building DNN-based PDE solvers with uniform frequency representation, bridging deep learning with established multiscale numerical methods.

Abstract

In this review, we survey the latest approaches and techniques developed to overcome the spectral bias towards low frequency of deep neural network learning methods in learning multiple-frequency solutions of partial differential equations. Open problems and future research directions are also discussed.
Paper Structure (29 sections, 1 theorem, 71 equations, 8 figures)

This paper contains 29 sections, 1 theorem, 71 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Preliminaries on DNN learning of PDE solutions
  3. Deep neural network (DNN)
  4. DNN based methods for solving PDEs
  5. PDE residual based learning -- physics-informed neural network (PINN)
  6. Variational energy based Deep-Ritz method
  7. Spectral bias of deep neural networks
  8. Empirical evidence of spectral bias
  9. Qualitative theory of spectral bias
  10. Linear quantitative theory of spectral bias
  11. Frequency shifting approach
  12. Frequency scaling approach
  13. Multiscale DNN (MscaleDNN)
  14. The idea of Multiscale DNN
  15. Spectral bias reduction property of the MscaleDNN, and selection of scales.
  16. ...and 14 more sections

Key Result

Corollary 1

Under mild assumptions and if $\sigma_b\gg 1$ and $\sigma=\mathrm{ReLU}$, then the dynamics eq..DynamicsFiniteWidth satisfies the following expression, where $\phi\in \mathcal{S}(\mathbb{R}^d)$ is a test function and the LFP operator reads as where the expectations are taken w.r.t. initial parameter distribution, $r = \lVert\bm{w}\rVert$, $\mathcal{F}[\cdot]$ indicates Fourier transform.

Figures (8)

  • Figure 1: 1-d frequency principle. (a) $f^{*}(x)$. (b) : $||\hat{f}^{*}(k)||$. (c) $\Delta_{F}(k)$ of three important frequencies (indicated by red dots in the inset of (b)) against different training epochs (horizontal axis). Here the y-axis represents the 3 frequencies, and the colorbar indicates the loss values.
  • Figure 2: Illustration of a PhaseDNN structure.
  • Figure 3: Results demonstrating the efficacy of using PhaseDNN compared to a normal fully-connected DNN. (a) Numerical and exact solution of Helmholtz equation using the PhaseDNN. (b) Failure of a normal fully-connected DNN.
  • Figure 4: Illustration of two MscaleDNN structures liu2020multi.
  • Figure 5: Comparison of a MscaleDNN and a vanilla DNN for an oscillatory solution of the Poisson equation at y=-0.33. (a) Numerical and exact solution of Poisson equation using MscaleDNN. (b) Failure of a vanilla fully-connected DNN (FC-DNN).
  • ...and 3 more figures

Theorems & Definitions (1)

  • Corollary 1: Luo et al., (2022) luo2022exact: LFP operator for ReLU