Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators

Weiye Gan, Yicheng Li, Qian Lin, Zuoqiang Shi

TL;DR

The paper addresses spectral bias in physics-informed neural networks by developing a Neural Tangent Kernel (NTK) framework for deep nets trained with a loss informed by differential operators. It derives that the NTK for the physics-informed loss factors as $K_{\mathcal{T}}^{NT}(x,x') = \mathcal{T}_x\mathcal{T}_{x'}K^{NT}(x,x')$, linking the operator-informed dynamics to the standard NTK, and proves convergence properties of the NTK both at initialization and during training in the wide-network limit. The results show that, in most cases, the differential operator in the loss does not induce a faster eigenvalue decay nor a stronger spectral bias, supported by perturbation arguments and empirical experiments. The findings suggest that improving PINP performance from a spectral-bias perspective is better achieved by balancing the loss components rather than adding higher-order differential operators, with implications for training efficiency and generalization in PDE-solving contexts.

Abstract

Spectral bias is a significant phenomenon in neural network training and can be explained by neural tangent kernel (NTK) theory. In this work, we develop the NTK theory for deep neural networks with physics-informed loss, providing insights into the convergence of NTK during initialization and training, and revealing its explicit structure. We find that, in most cases, the differential operators in the loss function do not induce a faster eigenvalue decay rate and stronger spectral bias. Some experimental results are also presented to verify the theory.

Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators

TL;DR

The paper addresses spectral bias in physics-informed neural networks by developing a Neural Tangent Kernel (NTK) framework for deep nets trained with a loss informed by differential operators. It derives that the NTK for the physics-informed loss factors as , linking the operator-informed dynamics to the standard NTK, and proves convergence properties of the NTK both at initialization and during training in the wide-network limit. The results show that, in most cases, the differential operator in the loss does not induce a faster eigenvalue decay nor a stronger spectral bias, supported by perturbation arguments and empirical experiments. The findings suggest that improving PINP performance from a spectral-bias perspective is better achieved by balancing the loss components rather than adding higher-order differential operators, with implications for training efficiency and generalization in PDE-solving contexts.

Abstract

Spectral bias is a significant phenomenon in neural network training and can be explained by neural tangent kernel (NTK) theory. In this work, we develop the NTK theory for deep neural networks with physics-informed loss, providing insights into the convergence of NTK during initialization and training, and revealing its explicit structure. We find that, in most cases, the differential operators in the loss function do not induce a faster eigenvalue decay rate and stronger spectral bias. Some experimental results are also presented to verify the theory.

Paper Structure

This paper contains 17 sections, 22 theorems, 154 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Continuously Differential Function Spaces
  4. Settings of Neural Network
  5. Training Dynamics
  6. NTK Theory
  7. The random feature and neural tangent kernel
  8. Convergnce of NNK
  9. Convergence at initialization
  10. Main Results
  11. Convergence at initialization
  12. Convergence in training
  13. Impact on the Spectrum
  14. Experiments
  15. Conclusion
  16. ...and 2 more sections

Key Result

Lemma 2.1

Let $T_0\subset\mathbb{R}^{n_0}$, $T_1\subset\mathbb{R}^{n_1}$ be two compact sets. Let $\varphi:T_0\rightarrow T_1$ and $\psi:T_1\rightarrow \mathbb{R}^{n_2}$ be two $C^{k,1}$ maps. Then, $\psi\circ\varphi:T_0\rightarrow \mathbb{R}^{n_2}$ is also a $C^{k,1}$ map. And there exists a constant $C$ onl

Figures (2)

  • Figure 1: Eigenvalues of $K_{\mathcal{T},\theta}(X,X)$ at initialization
  • Figure 2: $\mathcal{L}_1(u)$, $\mathcal{L}_2(u)$ and $\mathcal{L}_3(u)$ for different $a$ in training

Theorems & Definitions (38)

  • Lemma 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4: Proposition 34 in li2024_EigenvalueDecay
  • Lemma 2.5: hanin2021_RandomNeural
  • Theorem 3.1: Convergence of initial function
  • proof
  • Theorem 3.2
  • proof
  • Proposition 3.3
  • ...and 28 more