An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks

TL;DR

Residual-based Smote is introduced, an innovative local adaptive sampling technique tailored to improve the performance of Physics-Informed Neural Networks through imbalanced learning strategies and is positioned as a robust and resource-efficient solution for solving complex partial differential equations.

Abstract

This paper introduces Residual-based Smote (RSmote), an innovative local adaptive sampling technique tailored to improve the performance of Physics-Informed Neural Networks (PINNs) through imbalanced learning strategies. Traditional residual-based adaptive sampling methods, while effective in enhancing PINN accuracy, often struggle with efficiency and high memory consumption, particularly in high-dimensional problems. RSmote addresses these challenges by targeting regions with high residuals and employing oversampling techniques from imbalanced learning to refine the sampling process. Our approach is underpinned by a rigorous theoretical analysis that supports the effectiveness of RSmote in managing computational resources more efficiently. Through extensive evaluations, we benchmark RSmote against the state-of-the-art Residual-based Adaptive Distribution (RAD) method across a variety of dimensions and differential equations. The results demonstrate that RSmote not only achieves or exceeds the accuracy of RAD but also significantly reduces memory usage, making it particularly advantageous in high-dimensional scenarios. These contributions position RSmote as a robust and resource-efficient solution for solving complex partial differential equations, especially when computational constraints are a critical consideration.

Figures (11)

• Figure 1: Evolution of the imbalanced phenomenon in PDE dynamics over iterations. The figure displays the PDE dynamic, PDE residuals, and histograms of the residuals for three different iterations (100, 500, and 1000). The left column shows the comparison between the predicted solution from the PINN and the numerical solution. The middle column visualizes the residuals of the PDE at each point, and the right column presents the distribution of these residuals splited by a fixed threshold. The plots demonstrate how the imbalanced phenomenon evolves as training progresses, with the residuals showing different patterns at each stage. Here the split value is chosen as 0.1.
• Figure 2: (a) Compute the absolute residuals. (b) Sort the residuals in descending order and use a ratio, $\lambda$, to split the dataset. The horizontal axis represents the ratio value, while the vertical axis represents the residual values. (c) Select the positive and negative samples based on the ratio. (d) Apply SMOTE to perform resampling and create a new training dataset.
• Figure 3: Loss curves for Laplace Equation. Red line: Mean values of RAD-50000 method; Blue line: Mean values of RAD-100000 method; Green line: Mean values of RSmote method. The shaded areas represent the corresponding standard deviations.
• Figure 4: Solution fields for Laplace Equation. (a)-(c): ground truth, RAD solution and RSmote solution; (d)-(e): Absolute differences.
• Figure 5: Loss curves for Burgers’ Equation. Red line: Mean values of RAD-50000 method; Blue line: Mean values of RAD-100000 method; Green line: Mean values of RSmote method. The shaded areas represent the corresponding standard deviations.

Theorems & Definitions (6)

• proof
• Definition 1: VC-dimension abu1989vapnik
• Proposition 1
• proof
• Proposition 2
• Theorem 1