Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling

TL;DR

This work reframes PINN failure modes as propagation failures, where the correct PDE solution must propagate from boundary data to interior points during training. It introduces Retain-Resample-Release (R3) sampling to selectively accumulate collocation points in high residual regions while keeping computation light, and extends it causally for time-dependent PDEs. Theoretical properties (Retain, Resample, Release) are proven, and a causal version (Causal R3) is proposed to respect temporal propagation. Empirically, R3 and Causal R3 consistently outperform baseline sampling strategies across several PDEs, achieving lower errors with fewer collocation points and faster convergence, thereby enabling more efficient PINN training in challenging problems.

Abstract

Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing the "failure modes" of PINNs, although a thorough understanding of the connection between PINN failure modes and sampling strategies is missing. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that training PINNs relies on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures, characterized by highly imbalanced PDE residual fields. To mitigate propagation failures, we propose a novel Retain-Resample-Release sampling (R3) algorithm that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of R3 sampling to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of R3 sampling and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.

Figures (53)

• Figure 1: PINN solutions for a simple ODE: $u_{xx} + k^2 u = 0$ ($k =20$) with the analytical solution, $u=A\sin(kx) + B\cos(kx)$. The boundary condition was set to $u(-\pi/2)=0$, and the PINN is trained with 1000 equispaced collocation points. We can see smooth propagation of the correct solution from the boundary point at $x=0$ to interior points ($x>0$) as we increase training iterations.
• Figure 2: Demonstration of propagation failure using skewness and kurtosis while solving the convection equation with $\beta=50$.
• Figure 3: Demonstration of the Exact PDE solution, the predicted PDE solution, and the PDE Residual Field for the convection equation with $\beta = 10$ (first three figures) and $\beta = 50$ (last three figures) respectively.
• Figure 4: Schematic to describe our proposed R3 sampling algorithm, where collocation points are incrementally accumulated in regions with high PDE residuals (shown as contour lines).
• Figure 5: Causal R3 uses a time-dependent causal gate for computing PDE loss and for sampling.

Theorems & Definitions (15)

• Remark 3.1
• Theorem 4.1: Accumulation Dynamics Theorem
• Theorem 4.2: Non-Empty Theorem
• Theorem 1.1
• proof
• Definition 3.1: Objective Function
• Definition 3.2: $\epsilon$-maximal Neighborhood
• Lemma 3.3: Population Properties
• proof
• Lemma 3.4: Entry Condition