Stratified Sampling Algorithms for Machine Learning Methods in Solving Two-scale Partial Differential Equations

TL;DR

Stratified sampling addresses the challenge of solving two-scale PDEs with localized and flat solution regions by mitigating vanishing gradients in physics-informed neural networks (PINNs). The authors develop a two-stage training approach using active and diminishing gradient zones to guide sample selection, introducing algorithms for IC and PDE interior sampling within strata. Numerical results on Advection, Fisher, and Zeldovich equations show that stratified sampling often yields more accurate traveling-wave solutions and faster convergence than classical uniform sampling, especially on large domains. This method enables scalable PINNs for multi-scale PDEs by improving optimization robustness and solution fidelity across large computational domains.

Abstract

Partial differential equations (PDEs) with multiple scales or those defined over sufficiently large domains arise in various areas of science and engineering and often present problems when approximating the solutions numerically. Machine learning techniques are a relatively recent method for solving PDEs. Despite the increasing number of machine learning strategies developed to approximate PDEs, many remain focused on relatively small domains. When scaling the equations, a large domain is naturally obtained, especially when the solution exhibits multiscale characteristics. This study examines two-scale equations whose solution structures exhibit distinct characteristics: highly localized in some regions and significantly flat in others. These two regions must be adequately addressed over a large domain to approximate the solution more accurately. We focus on the vanishing gradient problem given by the diminishing gradient zone of the activation function over large domains and propose a stratified sampling algorithm to address this problem. We compare the uniform random classical sampling method over the entire domain and the proposed stratified sampling method. The numerical results confirm that the proposed method yields more accurate and consistent solutions than classical methods.

Figures (14)

• Figure 1: Schematic representation of the overall architecture of the model, considering the trial solution $U_{T}$ as a function of the input $(x,t)$ obtained from the neural network of $F$ hidden layers.
• Figure 2: Active and diminishing gradient zones of the sigmoid function $\sigma (x)=\frac{1}{1+e^{-x}}$.
• Figure 3: Exact initial condition $U(x, t_0)$ (dotted red line) given by Eq. $(\ref{['function_example1']})$ over the domain $\Omega_{x}=[-50,150]$ and its approximation $U_{T}(x)$ (solid blue line) determined by Eq. $(\ref{['NNs']})$ with three hidden layers and $60$ neurons per layer. The blue dots represent the sample points in the active gradient zones over $100,000$ epochs.
• Figure 4: (a) Convergence behavior of the loss function over the epochs. (b) Size of the sample sets for $U(x, t_0)$ given by Eq. $(\ref{['function_example1']})$ over the domain $\Omega_{x}=[-50,150]$ using the proposed sampling algorithm over $100,000$ epochs.
• Figure 5: Active and diminishing gradient zone schematic. The time domain is split into $N$ subintervals. The sample domain with the active gradient zone (solid red line centered between dotted red lines) is marked by blue boxes. Gray areas mark diminishing gradient zones. Sample points in blue subdomains train $U_{T}(x,t)$ given by Eq. (\ref{['NNs']}). Color available online.

Theorems & Definitions (18)

• Definition 1
• Proposition 1
• proof
• Corollary 1
• proof
• Definition 2
• Proposition 2
• proof
• Lemma 1
• proof