Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RoPINN: Region Optimized Physics-Informed Neural Networks

Haixu Wu, Huakun Luo, Yuezhou Ma, Jianmin Wang, Mingsheng Long

TL;DR

This paper proposes and theoretically studies a new training paradigm as region optimization of PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs.

Abstract

Physics-informed neural networks (PINNs) have been widely applied to solve partial differential equations (PDEs) by enforcing outputs and gradients of deep models to satisfy target equations. Due to the limitation of numerical computation, PINNs are conventionally optimized on finite selected points. However, since PDEs are usually defined on continuous domains, solely optimizing models on scattered points may be insufficient to obtain an accurate solution for the whole domain. To mitigate this inherent deficiency of the default scatter-point optimization, this paper proposes and theoretically studies a new training paradigm as region optimization. Concretely, we propose to extend the optimization process of PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs. A practical training algorithm, Region Optimized PINN (RoPINN), is seamlessly derived from this new paradigm, which is implemented by a straightforward but effective Monte Carlo sampling method. By calibrating the sampling process into trust regions, RoPINN finely balances optimization and generalization error. Experimentally, RoPINN consistently boosts the performance of diverse PINNs on a wide range of PDEs without extra backpropagation or gradient calculation. Code is available at this repository: https://github.com/thuml/RoPINN.

RoPINN: Region Optimized Physics-Informed Neural Networks

TL;DR

This paper proposes and theoretically studies a new training paradigm as region optimization of PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs.

Abstract

Physics-informed neural networks (PINNs) have been widely applied to solve partial differential equations (PDEs) by enforcing outputs and gradients of deep models to satisfy target equations. Due to the limitation of numerical computation, PINNs are conventionally optimized on finite selected points. However, since PDEs are usually defined on continuous domains, solely optimizing models on scattered points may be insufficient to obtain an accurate solution for the whole domain. To mitigate this inherent deficiency of the default scatter-point optimization, this paper proposes and theoretically studies a new training paradigm as region optimization. Concretely, we propose to extend the optimization process of PINNs from isolated points to their continuous neighborhood regions, which can theoretically decrease the generalization error, especially for hidden high-order constraints of PDEs. A practical training algorithm, Region Optimized PINN (RoPINN), is seamlessly derived from this new paradigm, which is implemented by a straightforward but effective Monte Carlo sampling method. By calibrating the sampling process into trust regions, RoPINN finely balances optimization and generalization error. Experimentally, RoPINN consistently boosts the performance of diverse PINNs on a wide range of PDEs without extra backpropagation or gradient calculation. Code is available at this repository: https://github.com/thuml/RoPINN.
Paper Structure (68 sections, 13 theorems, 71 equations, 10 figures, 9 tables, 1 algorithm)

This paper contains 68 sections, 13 theorems, 71 equations, 10 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. High-order regularization
  4. Variational formulation
  5. Method
  6. Region Optimization
  7. Generalization bound
  8. High-order PDE constraints
  9. Practical Algorithm
  10. Monte Carlo approximation
  11. Trust region calibration
  12. Balance between generalization and optimization
  13. Experiments
  14. Benchmarks
  15. Base models
  16. ...and 53 more sections

Key Result

Theorem 3.3

Suppose that the loss function $\mathcal{L}$ is $L$-Lipschitz-$\beta$-smooth for $\theta$. If we run stochastic gradient descent with step size $\alpha_{t}$ at the $t$-th step for $T$ iterations, we have that: (1) If $\mathcal{L}$ is convex for $\theta$ and $\alpha_{t}\leq\frac{2}{\beta}$, then $\ma

Figures (10)

  • Figure 1: Comparison between previous methods and ours. Previous point optimization methods train PINNs via the loss on selected points, which is different from our region optimization paradigm.
  • Figure 2: Optimization of canonical PINN Raissi2019PhysicsinformedNN on the 1D-Reaction under different region sizes. To highlight the region size change, we adopt the moving average over time and mark the temporal standard deviation with shadow. The steep training loss is caused by the learning difficulty of PDE.
  • Figure 3: Optimization of canonical PINN Raissi2019PhysicsinformedNN on the 1D-Reaction under different sample points.
  • Figure 4: Efficiency and model performance w.r.t. number of samples. Note that the default setting of RoPINN is just sampling one point, which will not bring extra gradient calculation costs.
  • Figure 5: Ablation study of RoPINN on different PDEs and diverse base models. rMSE is recorded.
  • ...and 5 more figures

Theorems & Definitions (32)

  • Definition 3.1
  • Theorem 3.3: Point optimization
  • Lemma 3.4
  • Theorem 3.5: Region optimization
  • proof
  • Corollary 3.6: Region optimization for first-order constraints
  • Example 3.7: Point optimization fails in optimizing with first-order constraints
  • Theorem 3.8: Convergence rate
  • Theorem 3.9: Gradient estimation error
  • Lemma 3.10: Trust region one-iteration approximation
  • ...and 22 more