Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs

TL;DR

The paper addresses the challenge of training neural operators for large-scale PDEs by identifying two bottlenecks in Fourier Neural Operators (FNO): the computational burden of high-resolution Fourier transforms and the sensitivity to the number of frequency modes. It introduces Incremental Fourier Neural Operator (iFNO), which progressively increases both the number of spectral coefficients $K$ and the input data resolution $R$ during training, guided by frequency-evolution criteria and an explained-ratio threshold, effectively acting as a dynamic spectral regularizer. Empirical results across Burgers, Darcy, Navier–Stokes, and Kolmogorov Flow show that iFNO achieves about 10% lower testing error with 20% fewer modes and around 30% faster training, illustrating improved generalization and efficiency. The approach enables high-resolution PDE simulations with limited labeled data and resources, and the authors discuss potential applications and broader impact in engineering and climate science, along with avenues for future physics-informed extensions.

Abstract

Fourier Neural Operators (FNO) offer a principled approach to solving challenging partial differential equations (PDE) such as turbulent flows. At the core of FNO is a spectral layer that leverages a discretization-convergent representation in the Fourier domain, and learns weights over a fixed set of frequencies. However, training FNO presents two significant challenges, particularly in large-scale, high-resolution applications: (i) Computing Fourier transform on high-resolution inputs is computationally intensive but necessary since fine-scale details are needed for solving many PDEs, such as fluid flows, (ii) selecting the relevant set of frequencies in the spectral layers is challenging, and too many modes can lead to overfitting, while too few can lead to underfitting. To address these issues, we introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases both the number of frequency modes used by the model as well as the resolution of the training data. We empirically show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.

Figures (10)

• Figure 1: Top: Fourier convolution operator in FNO. After the Fourier transform $\mathcal{F}$, the layer first truncates the full set of frequencies to the $K$ lowest ones using a dynamically set truncation before applying a learnable linear transformation (blue) and finally mapping the frequencies back to the linear space using the inverse FFT $\mathcal{F}^{-1}$. The previous method li_fourier_2021 picks a fixed $K$ throughout the entire training. Bottom: full iFNO architecture. Our model takes as input functions at different resolutions (discretizations). The operator consists of a lifting layer, followed by a series of iFNO blocks. The loss is used by our method to dynamically update the input resolution and the number of modes $K$ in the Spectral Convolutions. The incremental algorithm is detailed in section \ref{['sec:ifno']} and algorithm \ref{['alg:combined']}.
• Figure 2: The spectrum of Kolmogorov flow decays exponentially with the Kolmogorov scale of $5/3$ in the inverse cascade range.
• Figure 3: FNO with higher frequency modes captures smaller-scale structures in the turbulence. Prediction of the Kolmogorov flow by FNO with $K=10$ and $K=90$. Insufficient modes lead to overly strong dumping and it fails to capture small-scale structures in the turbulence of Kolmogorov flow.
• Figure 4: Frequency evolution of first and fourth Fourier convolution operators in FNO and iFNO during the training on Burgers' equation. We visualize FNO on the left figure and iFNO on the right figure for each layer. Each frequency strength $S_k$ is visualized across training. FNO is tuned with the optimal weight decay strength.
• Figure 5: Testing loss, training loss, and number of modes $K$ during the training of FNO and iFNO on Kolmogorov flow.

Theorems & Definitions (2)

• Definition 3.1: Neural operator
• Definition 3.2: Fourier convolution operator