Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence analysis of wide shallow neural operators within the framework of Neural Tangent Kernel

Xianliang Xu, Ye Li, Zhongyi Huang

TL;DR

The paper establishes that wide shallow neural operators trained with gradient-based methods under the NTK regime exhibit linear convergence to a global minimum when over-parameterized. By proving the strict positive definiteness and stability of the Gram matrices at infinite width, the authors show that both continuous-time gradient flow and discrete-time gradient descent converge rapidly, with weight updates remaining close to initialization. The results extend to Physics-Informed Neural Operators (PINNs), demonstrating that the training dynamics for operator learning with PDE residuals also enjoy NTK-style convergence, given appropriate width and learning-rate conditions. These findings provide theoretical guarantees for the trainability of neural operators in high-dimensional function spaces and offer guidance on network width and training strategies. They also lay groundwork for extending NTK-based convergence analyses to other operator architectures, such as FNOs and DeepONets, and for broader PINN contexts.

Abstract

Neural operators are aiming at approximating operators mapping between Banach spaces of functions, achieving much success in the field of scientific computing. Compared to certain deep learning-based solvers, such as Physics-Informed Neural Networks (PINNs), Deep Ritz Method (DRM), neural operators can solve a class of Partial Differential Equations (PDEs). Although much work has been done to analyze the approximation and generalization error of neural operators, there is still a lack of analysis on their training error. In this work, we conduct the convergence analysis of gradient descent for the wide shallow neural operators and physics-informed shallow neural operators within the framework of Neural Tangent Kernel (NTK). The core idea lies on the fact that over-parameterization and random initialization together ensure that each weight vector remains near its initialization throughout all iterations, yielding the linear convergence of gradient descent. In this work, we demonstrate that under the setting of over-parametrization, gradient descent can find the global minimum regardless of whether it is in continuous time or discrete time.

Convergence analysis of wide shallow neural operators within the framework of Neural Tangent Kernel

TL;DR

The paper establishes that wide shallow neural operators trained with gradient-based methods under the NTK regime exhibit linear convergence to a global minimum when over-parameterized. By proving the strict positive definiteness and stability of the Gram matrices at infinite width, the authors show that both continuous-time gradient flow and discrete-time gradient descent converge rapidly, with weight updates remaining close to initialization. The results extend to Physics-Informed Neural Operators (PINNs), demonstrating that the training dynamics for operator learning with PDE residuals also enjoy NTK-style convergence, given appropriate width and learning-rate conditions. These findings provide theoretical guarantees for the trainability of neural operators in high-dimensional function spaces and offer guidance on network width and training strategies. They also lay groundwork for extending NTK-based convergence analyses to other operator architectures, such as FNOs and DeepONets, and for broader PINN contexts.

Abstract

Neural operators are aiming at approximating operators mapping between Banach spaces of functions, achieving much success in the field of scientific computing. Compared to certain deep learning-based solvers, such as Physics-Informed Neural Networks (PINNs), Deep Ritz Method (DRM), neural operators can solve a class of Partial Differential Equations (PDEs). Although much work has been done to analyze the approximation and generalization error of neural operators, there is still a lack of analysis on their training error. In this work, we conduct the convergence analysis of gradient descent for the wide shallow neural operators and physics-informed shallow neural operators within the framework of Neural Tangent Kernel (NTK). The core idea lies on the fact that over-parameterization and random initialization together ensure that each weight vector remains near its initialization throughout all iterations, yielding the linear convergence of gradient descent. In this work, we demonstrate that under the setting of over-parametrization, gradient descent can find the global minimum regardless of whether it is in continuous time or discrete time.

Paper Structure

This paper contains 25 sections, 26 theorems, 328 equations.

Table of Contents

  1. Introduction
  2. Notations
  3. Preliminaries
  4. Continuous Time Analysis
  5. Discrete Time Analysis
  6. Physics-Informed Neural Operators
  7. Conclusion and Future Work
  8. Proof of Continuous Time Analysis
  9. Proof of Lemma 1
  10. Proof of Lemma 2
  11. Proof of Lemma 3
  12. Proof of Theorem 1
  13. Proof of Descrete Time Analysis
  14. Proof of Lemma 4
  15. Proof of Lemma 5
  16. ...and 10 more sections

Key Result

Lemma 1

If no two samples in $\{y_j\}_{j=1}^{n_2}$ are parallel and no two samples in $\{u_i\}_{i=1}^{n_1}$ are parallel, then $H^{\infty}$ and $\tilde{H}^{\infty}$ are strictly positive definite. We denote the least eigenvalue of $H^{\infty}$ and $\tilde{H}^{\infty}$ as $\lambda_0$ and $\tilde{\lambda}_0$

Theorems & Definitions (51)

  • Lemma 1
  • Remark 1
  • Lemma 2
  • Lemma 3
  • Theorem 1
  • Remark 2
  • Theorem 2
  • Lemma 4
  • Corollary 1
  • Lemma 5
  • ...and 41 more