Toward a Better Understanding of Fourier Neural Operators from a Spectral Perspective

TL;DR

This paper offers empirical insights into FNO's difficulty with large kernels through spectral analysis, and proposes SpecB-FNO to enhance the capture of non-dominant frequencies by adopting additional residual modules to learn from the previous ones' prediction residuals iteratively.

Abstract

In solving partial differential equations (PDEs), Fourier Neural Operators (FNOs) have exhibited notable effectiveness. However, FNO is observed to be ineffective with large Fourier kernels that parameterize more frequencies. Current solutions rely on setting small kernels, restricting FNO's ability to capture complex PDE data in real-world applications. This paper offers empirical insights into FNO's difficulty with large kernels through spectral analysis: FNO exhibits a unique Fourier parameterization bias, excelling at learning dominant frequencies in target data while struggling with non-dominant frequencies. To mitigate such a bias, we propose SpecB-FNO to enhance the capture of non-dominant frequencies by adopting additional residual modules to learn from the previous ones' prediction residuals iteratively. By effectively utilizing large Fourier kernels, SpecB-FNO achieves better prediction accuracy on diverse PDE applications, with an average improvement of 50%.

Figures (9)

• Figure 1: Energy density distribution of pixels with small to large features on Navier-Stokes ($\nu$ = 1e-5). In Fourier space, the energy distribution of the target data is more concentrated than in spatial space. Specifically, 1.2% of the pixels with larger features contain 99% of the energy in Fourier space. In contrast, the prediction residual has a more even energy distribution in Fourier space.
• Figure 2: NMSE spectrums on different PDE datasets. FNO's truncation frequency, k, is marked with a dotted line. The target energy reference is the energy spectrum of the target data, providing information on dominating frequencies. Two features of the diffusion-reaction equation (activator and inhibitor) are presented separately due to their different dominating frequencies.
• Figure 3: Illustration of SpecB-FNO with $T=1$.
• Figure 4: NMSE spectrums on Darcy flow with different stages of SpecB-FNO. The truncation frequency, k, is marked with a dotted line. In the initial stage, SpecB-FNO collapses to FNO.
• Figure 5: FNO architecture and designs for Fourier layers.