Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HEATNETs: Explainable Random Feature Neural Networks for High-Dimensional Parabolic PDEs

Kyriakos Georgiou, Gianluca Fabiani, Constantinos Siettos, Athanasios N. Yannacopoulos

TL;DR

This work addresses solving forward problems for high-dimensional time-dependent parabolic PDEs by introducing HEATNETs, a physics-informed RFNN whose random features are heat-kernel bases drawn from Green's functions. The authors prove an unbiased universal approximation with convergence rate $O(N^{-1/2})$ and couple this with importance sampling and PINN-based training to handle heat-kernel singularities and scale to dimensions up to $d=2000$; they report errors ranging from $1.0\times 10^{-5}$ to $1.0\times 10^{-7}$ for mid-range dimensions and $10^{-4}$ to $10^{-3}$ for the largest cases using up to 15,000 features. Numerical experiments span 1D to $d=2000$, demonstrating remarkable accuracy with relatively small feature counts and highlighting the explainability of HEATNETs through their grounding in the underlying PDE theory. The results suggest HEATNETs offer a transparent, efficient alternative to conventional PINNs for high-dimensional parabolic PDEs, with a well-defined mathematical foundation and practical scalability.

Abstract

We deal with the solution of the forward problem for high-dimensional parabolic PDEs with random feature (projection) neural networks (RFNNs). We first prove that there exists a single-hidden layer neural network with randomized heat-kernels arising from the fundamental solution (Green's functions) of the heat operator, that we call HEATNET, that provides an unbiased universal approximator to the solution of parabolic PDEs in arbitrary (high) dimensions, with the rate of convergence being analogous to the ${O}(N^{-1/2})$, where $N$ is the size of HEATNET. Thus, HEATNETs are explainable schemes, based on the analytical framework of parabolic PDEs, exploiting insights from physics-informed neural networks aided by numerical and functional analysis, and the structure of the corresponding solution operators. Importantly, we show how HEATNETs can be scaled up for the efficient numerical solution of arbitrary high-dimensional parabolic PDEs using suitable transformations and importance Monte Carlo sampling of the integral representation of the solution, in order to deal with the singularities of the heat kernel around the collocation points. We evaluate the performance of HEATNETs through benchmark linear parabolic problems up to 2,000 dimensions. We show that HEATNETs result in remarkable accuracy with the order of the approximation error ranging from $1.0E-05$ to $1.0E-07$ for problems up to 500 dimensions, and of the order of $1.0E-04$ to $1.0E-03$ for 1,000 to 2,000 dimensions, with a relatively low number (up to 15,000) of features.

HEATNETs: Explainable Random Feature Neural Networks for High-Dimensional Parabolic PDEs

TL;DR

This work addresses solving forward problems for high-dimensional time-dependent parabolic PDEs by introducing HEATNETs, a physics-informed RFNN whose random features are heat-kernel bases drawn from Green's functions. The authors prove an unbiased universal approximation with convergence rate and couple this with importance sampling and PINN-based training to handle heat-kernel singularities and scale to dimensions up to ; they report errors ranging from to for mid-range dimensions and to for the largest cases using up to 15,000 features. Numerical experiments span 1D to , demonstrating remarkable accuracy with relatively small feature counts and highlighting the explainability of HEATNETs through their grounding in the underlying PDE theory. The results suggest HEATNETs offer a transparent, efficient alternative to conventional PINNs for high-dimensional parabolic PDEs, with a well-defined mathematical foundation and practical scalability.

Abstract

We deal with the solution of the forward problem for high-dimensional parabolic PDEs with random feature (projection) neural networks (RFNNs). We first prove that there exists a single-hidden layer neural network with randomized heat-kernels arising from the fundamental solution (Green's functions) of the heat operator, that we call HEATNET, that provides an unbiased universal approximator to the solution of parabolic PDEs in arbitrary (high) dimensions, with the rate of convergence being analogous to the , where is the size of HEATNET. Thus, HEATNETs are explainable schemes, based on the analytical framework of parabolic PDEs, exploiting insights from physics-informed neural networks aided by numerical and functional analysis, and the structure of the corresponding solution operators. Importantly, we show how HEATNETs can be scaled up for the efficient numerical solution of arbitrary high-dimensional parabolic PDEs using suitable transformations and importance Monte Carlo sampling of the integral representation of the solution, in order to deal with the singularities of the heat kernel around the collocation points. We evaluate the performance of HEATNETs through benchmark linear parabolic problems up to 2,000 dimensions. We show that HEATNETs result in remarkable accuracy with the order of the approximation error ranging from to for problems up to 500 dimensions, and of the order of to for 1,000 to 2,000 dimensions, with a relatively low number (up to 15,000) of features.

Paper Structure

This paper contains 22 sections, 3 theorems, 68 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature review and state-of-the-art for high-dimensional PDEs
  3. Novelty and contribution of the proposed method
  4. Problem Statement and preliminaries
  5. Preliminaries: Potential representation and mild solutions
  6. Mild solutions
  7. HEATNETs: Shallow random feature/ projection neural networks with heat-kernels
  8. Choosing the randomization scheme
  9. Choosing the basis functions
  10. Implementation
  11. Dealing with the singularities
  12. Importance sampling for high-dimensions
  13. PINN implementation: architecture and training
  14. Model architecture and construction.
  15. Model training.
  16. ...and 7 more sections

Key Result

Proposition 1

There exists a shallow random feature neural network (RFNN), ${\cal R}(w, \phi)$, that can approximate arbitrarily close any function $u \in S$.

Figures (5)

  • Figure 1: The Random Feature Neural Network used in the HEATNETs framework: The model consists of a single hidden layer with $M = M_0+M_1$ nodes. Each node corresponds to a random feature with $\varphi^{(0)}_j(t,x), \varphi^{(1)}_j(t,x)$ given by \ref{['features-transformed']} or \ref{['features-is']}. The weights from the inputs $t \in [0,T], x\in \mathbb{R}^d$ are units, by construction, and the weights from the hidden layer to the output are $w = [w^{(0)} \,\,\ w^{(1)}]^T$.
  • Figure 2: HEATNETs results for \ref{['ex1']} on test grid. The RFNNs were constructed using $M_0=32$ and $M_1 =64$ features for the initial condition and the forcing term features, respectively and we consider 4,000 training points ($N_{PDE} = 3,000$ and $N_{IC} = 1,000$).
  • Figure 3: First row: Error metrics for the HEATNETs solution of \ref{['example-high-d']} for various time horizons, using $N_{PDE} =10,000, N_{IC} =2,048$ training points. First row: HEATNET with $M=1,500$ neurons (features) ($M_0 = 500, M_1 = 1,000$). Second row: HEATNET with $M = 8,000$ neurons (features) ($M_0 = 3,000, M_1 = 5,000$).
  • Figure 4: Median error metrics, $P_{50}$, 10/90 percentile bands, $[P_{10}, P_{90}]$, and interquartile range, $[P_{25}, P_{75}]$, for the HEATNETs solution of \ref{['example-high-d']}, for a time horizon $T =0.5$, using 600 training points ($N_{PDE} = 500, N_{IC} =100$) and $M=8,000$ neurons (features) in the hidden layer ($M_0 = 3,000, M_1 = 5,000$) with Sobol sampling. (Error metrics defined as in Fig. \ref{['fig:high-dim-PINN-graphs']}.)
  • Figure 5: HEATNET results for PDE \ref{['fig:high-dim-non-sep']}, using $M = 10,000$ features and $N_{PDE} = 15,000, N_{IC} = 3,000$.

Theorems & Definitions (8)

  • Definition 2.1: Mild solution
  • Proposition 1
  • proof
  • Remark 3.1
  • Proposition 2: Alternative sampling schemes
  • proof
  • Proposition 3
  • proof