Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tackling the Curse of Dimensionality with Physics-Informed Neural Networks

Zheyuan Hu, Khemraj Shukla, George Em Karniadakis, Kenji Kawaguchi

TL;DR

A new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs, called Stochastic Dimension Gradient Descent (SDGD), which decomposes a gradient of PDEs' and PINNs' residual into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs.

Abstract

The curse-of-dimensionality taxes computational resources heavily with exponentially increasing computational cost as the dimension increases. This poses great challenges in solving high-dimensional PDEs, as Richard E. Bellman first pointed out over 60 years ago. While there has been some recent success in solving numerically partial differential equations (PDEs) in high dimensions, such computations are prohibitively expensive, and true scaling of general nonlinear PDEs to high dimensions has never been achieved. We develop a new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs. The new method, called Stochastic Dimension Gradient Descent (SDGD), decomposes a gradient of PDEs into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs. We prove theoretically the convergence and other desired properties of the proposed method. We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton-Jacobi-Bellman (HJB) and the Schrödinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach. Notably, we solve nonlinear PDEs with nontrivial, anisotropic, and inseparable solutions in 100,000 effective dimensions in 12 hours on a single GPU using SDGD with PINNs. Since SDGD is a general training methodology of PINNs, it can be applied to any current and future variants of PINNs to scale them up for arbitrary high-dimensional PDEs.

Tackling the Curse of Dimensionality with Physics-Informed Neural Networks

TL;DR

A new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs, called Stochastic Dimension Gradient Descent (SDGD), which decomposes a gradient of PDEs' and PINNs' residual into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs.

Abstract

The curse-of-dimensionality taxes computational resources heavily with exponentially increasing computational cost as the dimension increases. This poses great challenges in solving high-dimensional PDEs, as Richard E. Bellman first pointed out over 60 years ago. While there has been some recent success in solving numerically partial differential equations (PDEs) in high dimensions, such computations are prohibitively expensive, and true scaling of general nonlinear PDEs to high dimensions has never been achieved. We develop a new method of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional PDEs. The new method, called Stochastic Dimension Gradient Descent (SDGD), decomposes a gradient of PDEs into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs. We prove theoretically the convergence and other desired properties of the proposed method. We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton-Jacobi-Bellman (HJB) and the Schrödinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach. Notably, we solve nonlinear PDEs with nontrivial, anisotropic, and inseparable solutions in 100,000 effective dimensions in 12 hours on a single GPU using SDGD with PINNs. Since SDGD is a general training methodology of PINNs, it can be applied to any current and future variants of PINNs to scale them up for arbitrary high-dimensional PDEs.
Paper Structure (38 sections, 7 theorems, 62 equations, 2 figures, 5 tables, 3 algorithms)

This paper contains 38 sections, 7 theorems, 62 equations, 2 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Physics-Informed Machine Learning
  4. Machine Learning PDEs Solvers in High-Dimensions
  5. Stochastic Gradient Descent
  6. Method
  7. Physics-Informed Neural Networks (PINNs)
  8. Methodology for High-Dimensional PDEs
  9. Introductory Case of the High-Dimensional Poisson's Equation
  10. General Case
  11. Extensions: Gradient Accumulation and Parallel Computing
  12. Further Speed-up via Simultaneous Sampling in Forward and Backward Passes
  13. Additional Speed-up via Sampling Only Once
  14. Theory
  15. SDGD is Unbiased
  16. ...and 23 more sections

Key Result

Theorem 4.1

The stochastic gradients $\text{grad}_I(\theta)$ and $\text{grad}_{I,J}(\theta)$ in our Algorithms algo:1 and algo:2, respectively, parameterized by index sets $I, J$, are an unbiased estimator of the full-batch gradient $\text{grad}(\theta)$ using all PDE terms, i.e., the expected values of these e

Figures (2)

  • Figure 1: SDGD convergence curve for the Poisson, Allen-Cahn, and Sine-Gordon PDEs in 100, 1K, 10K, and 100K dimensions.
  • Figure 2: Coupled quantum harmonic oscillator (CQHO) problem: convergence curve of our algorithm (gradient accumulation) under different dimensions.

Theorems & Definitions (22)

  • Theorem 4.1
  • proof
  • Theorem 4.2
  • proof
  • Definition 4.1
  • Remark 1
  • Lemma 4.1
  • proof
  • Remark 2
  • Definition 4.2
  • ...and 12 more