Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feature Mapping in Physics-Informed Neural Networks (PINNs)

Chengxi Zeng, Tilo Burghardt, Alberto M Gambaruto

Abstract

In this paper, the training dynamics of PINNs with a feature mapping layer via the limiting Conjugate Kernel and Neural Tangent Kernel is investigated, shedding light on the convergence of PINNs; Although the commonly used Fourier-based feature mapping has achieved great success, we show its inadequacy in some physics scenarios. Via these two scopes, we propose conditionally positive definite Radial Basis Function as a better alternative. Lastly, we explore the feature mapping numerically in wide neural networks. Our empirical results reveal the efficacy of our method in diverse forward and inverse problem sets. Composing feature functions is found to be a practical way to address the expressivity and generalisability trade-off, viz., tuning the bandwidth of the kernels and the surjectivity of the feature mapping function. This simple technique can be implemented for coordinate inputs and benefits the broader PINNs research.

Feature Mapping in Physics-Informed Neural Networks (PINNs)

Abstract

In this paper, the training dynamics of PINNs with a feature mapping layer via the limiting Conjugate Kernel and Neural Tangent Kernel is investigated, shedding light on the convergence of PINNs; Although the commonly used Fourier-based feature mapping has achieved great success, we show its inadequacy in some physics scenarios. Via these two scopes, we propose conditionally positive definite Radial Basis Function as a better alternative. Lastly, we explore the feature mapping numerically in wide neural networks. Our empirical results reveal the efficacy of our method in diverse forward and inverse problem sets. Composing feature functions is found to be a practical way to address the expressivity and generalisability trade-off, viz., tuning the bandwidth of the kernels and the surjectivity of the feature mapping function. This simple technique can be implemented for coordinate inputs and benefits the broader PINNs research.
Paper Structure (50 sections, 3 theorems, 69 equations, 18 figures, 8 tables)

This paper contains 50 sections, 3 theorems, 69 equations, 18 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background and Prior theories
  3. Physics-Informed Neural Network
  4. Feature Mapping in PINNs
  5. Prior Theories
  6. Limitation of the Fourier Features
  7. Training dynamics of PINNs
  8. Theory Settings
  9. Proposed feature mapping method
  10. Compact support RBF
  11. Conditionally positive definite RBF
  12. Adding Surjectivity to RBF
  13. Empirical Results
  14. Experimental Setup
  15. Forward Problems
  16. ...and 35 more sections

Key Result

Lemma 2.1

Consider a randomly sampled and normalised input $\mathbf{x}=[x_1, x_2, \cdots, x_n]^T$, $x \in [0, 1]^d$, and its corresponding features in $\Phi: \mathbb{R}^d \rightarrow \mathbb{R}^m = [\varphi(x_1), \varphi(x_2), \cdots, \varphi(x_n)]^T$, let the feature mapping function $\varphi(\mathbf{x}) = s

Figures (18)

  • Figure 1: (a) Fourier based Positional Encoding shows inadequate generalisation at the discontinuity in Burgers' Equation; (b) Random Fourier Features fail at high dimensional Poisson Equation. Error on nD Poisson equation from 1 to 10 dimensions cases (left), and a more realistic setting with uneven sampling on each dimension (right). The experiments are repeated 3 times with different random seeds, and the variances are highlighted in shades.
  • Figure 2: (a) The bandwidth of the kernel can be controlled by the compact support radius of the RBFs; (b) The surjectivity can be adjusted by composing auxiliary Fourier features to the RBFs.
  • Figure 3: (a) Although the optimal value of $\xi$ varies, narrower kernel bandwidths are generally preferred in some PDEs. (b) Adding too much surjectivity to RBF is mostly disadvantageous, but it can bring extra performance improvements in a few situations.
  • Figure 4: Ablation study on different number of RBFs
  • Figure 5: Ablation study on different number of polynomials
  • ...and 13 more figures

Theorems & Definitions (7)

  • Lemma 2.1
  • proof
  • Theorem 3.1: Propagation of the Conjugate Kernel
  • proof
  • Theorem 3.2: Evolution of the NTK with CK
  • proof
  • Remark