Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Quadrature Rules for Solving PDEs using Neural Networks

Jamie M. Taylor, David Pardo

TL;DR

The paper addresses numerical integration challenges in neural PDE solvers, focusing on the DRM for Poisson-type problems. It shows that deterministic and biased stochastic quadrature can mislead solutions, and demonstrates that unbiased, high-order stochastic quadrature on low-dimensional integration meshes yields substantially faster convergence at comparable cost. It introduces new unbiased quadrature rules on triangular and tetrahedral meshes, enabling flexible meshing in complex geometries, and shows that convergence is ultimately limited by the variance of the stochastic gradient rather than the loss itself. The work highlights that accurate gradient integration is crucial for gradient-based optimisation and provides practical guidance and tools for variance reduction in low-dimensional neural PDE solvers.

Abstract

We examine the challenges associated with numerical integration when applying Neural Networks to solve Partial Differential Equations (PDEs). We specifically investigate the Deep Ritz Method (DRM), chosen for its practical applicability and known sensitivity to integration inaccuracies. Our research demonstrates that both standard deterministic integration techniques and biased stochastic quadrature methods can lead to incorrect solutions. In contrast, employing high-order, unbiased stochastic quadrature rules defined on integration meshes in low dimensions is shown to significantly enhance convergence rates at a comparable computational expense with respect to low-order methods like Monte Carlo. Additionally, we introduce novel stochastic quadrature approaches designed for triangular and tetrahedral mesh elements, offering increased adaptability for handling complex geometric domains. We highlight that the variance inherent in the stochastic gradient acts as a bottleneck for convergence. Furthermore, we observe that for gradient-based optimisation, the crucial factor is the accurate integration of the gradient, rather than just minimizing the quadrature error of the loss function itself.

Stochastic Quadrature Rules for Solving PDEs using Neural Networks

TL;DR

The paper addresses numerical integration challenges in neural PDE solvers, focusing on the DRM for Poisson-type problems. It shows that deterministic and biased stochastic quadrature can mislead solutions, and demonstrates that unbiased, high-order stochastic quadrature on low-dimensional integration meshes yields substantially faster convergence at comparable cost. It introduces new unbiased quadrature rules on triangular and tetrahedral meshes, enabling flexible meshing in complex geometries, and shows that convergence is ultimately limited by the variance of the stochastic gradient rather than the loss itself. The work highlights that accurate gradient integration is crucial for gradient-based optimisation and provides practical guidance and tools for variance reduction in low-dimensional neural PDE solvers.

Abstract

We examine the challenges associated with numerical integration when applying Neural Networks to solve Partial Differential Equations (PDEs). We specifically investigate the Deep Ritz Method (DRM), chosen for its practical applicability and known sensitivity to integration inaccuracies. Our research demonstrates that both standard deterministic integration techniques and biased stochastic quadrature methods can lead to incorrect solutions. In contrast, employing high-order, unbiased stochastic quadrature rules defined on integration meshes in low dimensions is shown to significantly enhance convergence rates at a comparable computational expense with respect to low-order methods like Monte Carlo. Additionally, we introduce novel stochastic quadrature approaches designed for triangular and tetrahedral mesh elements, offering increased adaptability for handling complex geometric domains. We highlight that the variance inherent in the stochastic gradient acts as a bottleneck for convergence. Furthermore, we observe that for gradient-based optimisation, the crucial factor is the accurate integration of the gradient, rather than just minimizing the quadrature error of the loss function itself.

Paper Structure

This paper contains 36 sections, 4 theorems, 116 equations, 11 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model Problems Using the Deep Ritz Method (DRM)
  3. The Deep Ritz Method.
  4. Architectures.
  5. Discretisation of the loss function.
  6. Deterministic quadrature
  7. Stochastic quadrature
  8. Monte Carlo methods
  9. Gauss-type stochastic quadrature on integration meshes
  10. Stochastic Gradient-based Methods
  11. Bias
  12. Unbiased stochastic quadrature rules
  13. Quadrature rules from the literature
  14. Order-0 unbiased quadrature rule ($P_0$)
  15. Order-1 unbiased quadrature rule ($P_1$).
  16. ...and 21 more sections

Key Result

Proposition A.1

Let $T$ be the reference triangle in 2D and reference tetrahedron in 3D. For the rules $P_2^{\text{tri}}$ in 2D and $P_2^\text{tet}$ in 3D, there exists a constant $C>0$ such that, for all $f\in C^3(\bar{T})$,

Figures (11)

  • Figure 1: Results of the DRM using deterministic quadrature.
  • Figure 2: Illustration of the master element and a mesh-based quadrature rule.
  • Figure 3: Comparison of results using biased ($P_1^b$) and unbiased ($P_1$) order-one rules.
  • Figure 4: Illustration of $W_h$ for $h=\frac{1}{4}$.
  • Figure 5: The triangular and tetrahedral decompositions of $[0,1]^d$.
  • ...and 6 more figures

Theorems & Definitions (9)

  • Remark 4.1
  • Proposition A.1
  • proof
  • Proposition B.1
  • proof
  • Proposition B.2
  • proof
  • Proposition B.3
  • proof