Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks

Zheyuan Hu, Zekun Shi, George Em Karniadakis, Kenji Kawaguchi

TL;DR

This work addresses the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE), and opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

Abstract

Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE). Starting with the second-order high-dimensional PDEs ubiquitous in scientific computing, HTE transforms the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach alleviates the computational bottleneck via Taylor-mode automatic differentiation and significantly reduces memory consumption from the Hessian matrix to HVP. We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variance among dimensions. We further extend HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks

TL;DR

This work addresses the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE), and opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

Abstract

Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE). Starting with the second-order high-dimensional PDEs ubiquitous in scientific computing, HTE transforms the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach alleviates the computational bottleneck via Taylor-mode automatic differentiation and significantly reduces memory consumption from the Hessian matrix to HVP. We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variance among dimensions. We further extend HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.
Paper Structure (28 sections, 8 theorems, 40 equations, 2 figures, 5 tables)

This paper contains 28 sections, 8 theorems, 40 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Preliminaries
  5. HTE for High-Dimensional Second-Order Parabolic PDEs
  6. Methodology
  7. Discussion on the Bias and Unbiased HTE Versions
  8. HTE Implementation Detail
  9. Comparison with SDGD
  10. SDGD as a Special Case of HTE
  11. Variance Comparison between HTE and SDGD
  12. Applications to Biharmonic Equations
  13. Applicability of HTE
  14. HTE's Application to Other Neural PDE Solvers
  15. Discussion on HTE's Applicable PDE Types
  16. ...and 13 more sections

Key Result

Theorem 3.1

The loss $L_{\text{HTE}}(\theta)$ in equation (eq:HTE) converges almost surely (a.s.) to the exact PINN loss $L_{\text{PINN}}(\theta)$ in equation (eq:PINN), as $V \rightarrow \infty$, i.e., The loss $L_{\text{HTE, unbiased}}(\theta)$ in equation (eq:HTE_unbiased) is an unbiased estimator for the exact PINN loss $L_{\text{PINN}}(\theta)$ in equation (eq:PINN), i.e.,

Figures (2)

  • Figure :
  • Figure :

Theorems & Definitions (18)

  • Theorem 3.1
  • proof
  • Theorem 3.2
  • proof
  • Remark 1
  • Theorem 3.3
  • proof
  • Remark 2
  • Theorem 3.4
  • proof
  • ...and 8 more