Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orthogonal greedy algorithm for linear operator learning with shallow neural network

Ye Lin, Jiwei Jia, Young Ju Lee, Ran Zhang

TL;DR

This work addresses learning linear operators associated with PDE solution operators by focusing on kernel (Green's function) estimation. It introduces two orthogonal greedy algorithm (OGA)-based frameworks operating on shallow neural-network dictionaries, including a data-driven semi-inner product and a point-wise kernel estimation variant (PW-OGA) to improve convergence. The authors establish convergence guarantees in the semi-norm, derive rate estimates that align with optimal shallow-network bounds, and demonstrate through extensive 1D–3D PDE experiments that the proposed methods achieve orders-of-magnitude improvements over baselines such as GL, FNO, and DON. The results highlight the practicality and interpretability of kernel-based operator learning with shallow networks, while revealing nuances such as warm-up behavior and overfitting in PW-OGA, guiding future developments in efficient PDE operator learning. Overall, the paper provides a theoretically grounded and empirically validated path to accurate, data-efficient operator learning and Green's-function estimation using shallow models.

Abstract

Greedy algorithms, particularly the orthogonal greedy algorithm (OGA), have proven effective in training shallow neural networks for fitting functions and solving partial differential equations (PDEs). In this paper, we extend the application of OGA to the tasks of linear operator learning, which is equivalent to learning the kernel function through integral transforms. Firstly, a novel greedy algorithm is developed for kernel estimation rate in a new semi-inner product, which can be utilized to approximate the Green's function of linear PDEs from data. Secondly, we introduce the OGA for point-wise kernel estimation to further improve the approximation rate, achieving orders of accuracy improvement across various tasks and baseline models. In addition, we provide a theoretical analysis on the kernel estimation problem and the optimal approximation rates for both algorithms, establishing their efficacy and potential for future applications in PDEs and operator learning tasks.

Orthogonal greedy algorithm for linear operator learning with shallow neural network

TL;DR

This work addresses learning linear operators associated with PDE solution operators by focusing on kernel (Green's function) estimation. It introduces two orthogonal greedy algorithm (OGA)-based frameworks operating on shallow neural-network dictionaries, including a data-driven semi-inner product and a point-wise kernel estimation variant (PW-OGA) to improve convergence. The authors establish convergence guarantees in the semi-norm, derive rate estimates that align with optimal shallow-network bounds, and demonstrate through extensive 1D–3D PDE experiments that the proposed methods achieve orders-of-magnitude improvements over baselines such as GL, FNO, and DON. The results highlight the practicality and interpretability of kernel-based operator learning with shallow networks, while revealing nuances such as warm-up behavior and overfitting in PW-OGA, guiding future developments in efficient PDE operator learning. Overall, the paper provides a theoretically grounded and empirically validated path to accurate, data-efficient operator learning and Green's-function estimation using shallow models.

Abstract

Greedy algorithms, particularly the orthogonal greedy algorithm (OGA), have proven effective in training shallow neural networks for fitting functions and solving partial differential equations (PDEs). In this paper, we extend the application of OGA to the tasks of linear operator learning, which is equivalent to learning the kernel function through integral transforms. Firstly, a novel greedy algorithm is developed for kernel estimation rate in a new semi-inner product, which can be utilized to approximate the Green's function of linear PDEs from data. Secondly, we introduce the OGA for point-wise kernel estimation to further improve the approximation rate, achieving orders of accuracy improvement across various tasks and baseline models. In addition, we provide a theoretical analysis on the kernel estimation problem and the optimal approximation rates for both algorithms, establishing their efficacy and potential for future applications in PDEs and operator learning tasks.
Paper Structure (33 sections, 7 theorems, 75 equations, 18 figures, 9 tables, 4 algorithms)

This paper contains 33 sections, 7 theorems, 75 equations, 18 figures, 9 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Operator learning and kernel estimation
  3. Operator learning
  4. Linear operator learning and kernel estimation
  5. Nonlinear dictionary approximation with shallow neural network
  6. Nonlinear dictionary approximation
  7. Shallow neural network for function approximation in a Hilbert space
  8. Inner product in the linear operator learning setting
  9. Orthogonal greedy algorithm for function approximation
  10. Orthogonal greedy algorithm for kernel estimation
  11. Approximation rate analysis of OGA for kernel estimation
  12. Shallow neural network in practice
  13. Point-wise kernel estimation by OGA
  14. Numerical Results
  15. Numerical results for one-dimensional problems
  16. ...and 18 more sections

Key Result

Lemma 1

Assume that the data set $\{f_j\}^{N}_{j=1}$ contains at least $m_f$ linear independent functions, say $\{f_{j_\ell}\}^{m_f}_{\ell=1}$ with $j_\ell \in \{1,\cdots,N\}$. Then, the space $\mathcal{K}$ makes a finite dimensional compact set and the following bilinear form on $\mathcal{G}$ defines an in

Figures (18)

  • Figure 1: Schematic of Green Learning
  • Figure 2: exact kernel $G$ and its approximant $\hat{G}$ with $h=0.1$
  • Figure 3: (a) Kernel function. (b) Slices of kernel function.
  • Figure 4: Result of 1D Poisson problem: (a) The training dynamics of OGA method. (b) Exact Green's function. (c) Learned Green's function. (d) Absolute error $E_G$.
  • Figure 5: Result of 1D Helmholtz problem: (a) The training dynamics of OGA method. (b) Exact Green's function. (c) Learned Green's function. (d) Absolute error $E_G$.
  • ...and 13 more figures

Theorems & Definitions (13)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • Remark 1
  • Theorem 1
  • proof
  • Lemma 5
  • ...and 3 more