Iterative Training of Physics-Informed Neural Networks with Fourier-enhanced Features

TL;DR

IFeF-PINN, an algorithm for iterative training of PINNs with Fourier-enhanced features, empirically establish that Random Fourier Features enhance the expressive capacity of the network, enabling accurate approximation of high-frequency PDEs.

Abstract

Spectral bias, the tendency of neural networks to learn low-frequency features first, is a well-known issue with many training algorithms for physics-informed neural networks (PINNs). To overcome this issue, we propose IFeF-PINN, an algorithm for iterative training of PINNs with Fourier-enhanced features. The key idea is to enrich the latent space using high-frequency components through Random Fourier Features. This creates a two-stage training problem: (i) estimate a basis in the feature space, and (ii) perform regression to determine the coefficients of the enhanced basis functions. For an underlying linear model, it is shown that the latter problem is convex, and we prove that the iterative training scheme converges. Furthermore, we empirically establish that Random Fourier Features enhance the expressive capacity of the network, enabling accurate approximation of high-frequency PDEs. Through extensive numerical evaluation on classical benchmark problems, the superior performance of our method over state-of-the-art algorithms is shown, and the improved approximation across the frequency domain is illustrated.

Figures (10)

• Figure 1: Architecture of IFeF-PINN. The first part (in yellow) generates the nominal basis vectors, which are then extended via $\gamma_D$ generating random Fourier features $\psi_D$ (in green), and a linear combination of the extended basis (in blue) forms the approximated solution $u_{\omega,\theta}$.
• Figure 2: Boxplot of relative $L^2$-errors (log10 scale) for all methods on three low-frequency benchmarks with median, inter-quartile range (IQR), and mean (red diamonds).
• Figure 3: True solution, prediction and absolute error of baseline methods for Low-frequency Helmholtz equation
• Figure 4: True solution, prediction and absolute error of baseline methods for High-frequency Helmholtz equation
• Figure 5: Prediction of the network spectrum with an increasing number of Fourier features. The x-axis represents frequency, and the colorbar shows the normalized magnitude of the predicted solution at $t=0$. The colorbar is scaled accordingly from 0 to 1.

Theorems & Definitions (22)

• Proposition 1
• Remark 1
• Remark 2: On resampling strategies
• Proposition 2: Lipschitz Continuity of the Solution Map
• Theorem 1: Convergence to a stationary point
• Definition 1
• Theorem 2
• Corollary 1: Universal approximation
• proof
• Lemma 1: Sufficient Decrease