BO-SA-PINNs: Self-adaptive physics-informed neural networks based on Bayesian optimization for automatically designing PDE solvers

TL;DR

BO-SA-PINNs address the hyperparameter-tuning bottleneck in physics-informed neural networks by combining a three-stage workflow: (i) Bayesian optimization to automatically design architecture, learning rate, sampling distribution, and loss weights from pre-training, (ii) a global self-adaptive phase using EMA and RAR-D to refine weights and sampling during training, and (iii) a final L-BFGS refinement for stable convergence. A novel TG activation function is introduced to enhance expressivity and local feature capture. Across 2D Helmholtz, 2D Maxwell, Burgers, and high-dimensional Poisson problems, BO-SA-PINNs achieve higher accuracy with significantly fewer iterations and fewer total sampling points than baseline PINNs and outperform SA-PINNs in several cases, with ablation studies confirming the contribution of each component. The framework reduces manual tuning, improves training efficiency, and extends PINNs to more challenging PDEs, offering a practical route toward robust, automated PDE solvers, while also highlighting future work on preventing TG-related overfitting and enhancing global uncertainty quantification.

Abstract

Physics-informed neural networks (PINNs) is becoming a popular alternative method for solving partial differential equations (PDEs). However, they require dedicated manual modifications to the hyperparameters of the network, the sampling methods and loss function weights for different PDEs, which reduces the efficiency of the solvers. In this paper, we pro- pose a general multi-stage framework, i.e. BO-SA-PINNs to alleviate this issue. In the first stage, Bayesian optimization (BO) is used to select hyperparameters for the training process, and based on the results of the pre-training, the network architecture, learning rate, sampling points distribution and loss function weights suitable for the PDEs are automatically determined. The proposed hyperparameters search space based on experimental results can enhance the efficiency of BO in identifying optimal hyperparameters. After selecting the appropriate hyperparameters, we incorporate a global self-adaptive (SA) mechanism the second stage. Using the pre-trained model and loss information in the second-stage training, the exponential moving average (EMA) method is employed to optimize the loss function weights, and residual-based adaptive refinement with distribution (RAR-D) is used to optimize the sampling points distribution. In the third stage, L-BFGS is used for stable training. In addition, we introduce a new activation function that enables BO-SA-PINNs to achieve higher accuracy. In numerical experiments, we conduct comparative and ablation experiments to verify the performance of the model on Helmholtz, Maxwell, Burgers and high-dimensional Poisson equations. The comparative experiment results show that our model can achieve higher accuracy and fewer iterations in test cases, and the ablation experiments demonstrate the positive impact of every improvement.

Figures (5)

• Figure 1: Automated machine learning process
• Figure 2: TG activation function
• Figure 3: Entire process of BO-SA-PINNs; Stage 1 is selecting suitable hyperparameters, stage 2 is using self-adaptive mechanisms to adjust loss function weights and sampling points distribution, stage 3 is stable optimization.
• Figure 4: Train loss and test loss of TG activation function (TG), $tanh(x)$ (Tanh), $sin(x)$ (Sin) and $e^{-\frac{x^2}{2}}$ (Gaus); R denotes residual loss and B denotes boundary loss
• Figure 5: Predicted solution, true solution, and error of 2D Maxwell problem of BO-SA-PINN in $f=300$MHz, $\varepsilon_s=1, \varepsilon_c=1.5$, including real part and imaginary part