Annealed adaptive importance sampling method in PINNs for solving high dimensional partial differential equations

TL;DR

This work tackles the difficulty of training PINNs on high-dimensional and singular PDEs by introducing Annealed Adaptive Importance Sampling (AAIS) to drive adaptive resampling of collocation points. By modeling the PDE residual distribution with finite mixtures (Gaussian or Student's t) and employing an EM-based update guided by an annealed temperature ladder, AAIS-enhanced PINNs concentrate training resources where the residual is large, improving accuracy and efficiency. The authors also propose a simple resampling framework that maintains a fixed training set while injecting adaptive points according to the residual density, and demonstrate substantial gains over uniform sampling, with AAIS-t often outperforming AAIS-g and RAD in high-dimensional problems. The approach is versatile, scalable, and compatible with various PINN formulations, offering a practical path toward solving complex high-dimensional PDEs and informing future extensions to inverse problems and parallel implementations.

Abstract

Physics-informed neural networks (PINNs) have emerged as powerful tools for solving a wide range of partial differential equations (PDEs). However, despite their user-friendly interface and broad applicability, PINNs encounter challenges in accurately resolving PDEs, especially when dealing with singular cases that may lead to unsatisfactory local minima. To address these challenges and improve solution accuracy, we propose an innovative approach called Annealed Adaptive Importance Sampling (AAIS) for computing the discretized PDE residuals of the cost functions, inspired by the Expectation Maximization algorithm used in finite mixtures to mimic target density. Our objective is to approximate discretized PDE residuals by strategically sampling additional points in regions with elevated residuals, thus enhancing the effectiveness and accuracy of PINNs. Implemented together with a straightforward resampling strategy within PINNs, our AAIS algorithm demonstrates significant improvements in efficiency across a range of tested PDEs, even with limited training datasets. Moreover, our proposed AAIS-PINN method shows promising capabilities in solving high-dimensional singular PDEs. The adaptive sampling framework introduced here can be integrated into various PINN frameworks.

Figures (35)

• Figure 1: Exact solution for Poisson problems with one peak in \ref{['pde:Poisson1Peak']}.
• Figure 2: $L^2$ relative error and $L^\infty$ error during training for one peak Poisson equation with training schedule of lbfgs 1000 epochs each versus different searching points $N_S$. Left: the $L^\infty$ error $e_\infty(u(\cdot;\theta))$. Right: the relative $L^2$ error $e_r(u(\cdot;\theta))$.
• Figure 3: Residual $\mathcal{Q}$ for Uni methods for training schedule of lbfgs 1000 epochs where $N_S=7000$.
• Figure 4: Residual $\mathcal{Q}$ and training datasets for RAD, AAIS-g and AAIS-t for training schedule of lbfgs 1000 epochs after 9-th training. Left column: RAD. Middle column: AAIS-g. Right column: AAIS-t. $\mathcal{S}_j$ means the nodes sampled from the dataset used in $(j-1)$-th iteration and $\mathcal{D}$ is the adaptive sampling nodes from the residual $\mathcal{Q}$ correspondingly.
• Figure 5: Profiles of absolute error and neural network solutions for one peak Poisson equation with training schedule of 1000 epochs for lbfgs. First row: numerical solutions. Second row: absolute error. For Uni, the solution is generated after 5th iteration. For other three sampling methods, the solutions are from the 10th iteration.

Theorems & Definitions (2)

• Remark 3.1
• Remark 4.1