Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programming

TL;DR

This work tackles the ill-conditioning and convergence challenges of Physics-Informed Neural Networks (PINNs) when solving complex PDEs by introducing trSQP-PINN, a hard-constrained optimization framework based on trust-region Sequential Quadratic Programming. It replaces soft penalties with a linear-quadratic subproblem that enforces PDE constraints through linearized equations and a trust-region, while estimating second-order information via quasi-Newton updates (damped BFGS or SR1) and guiding steps with a merit function. A simple pretraining step initializes the network near the constraint manifold, reducing data requirements and improving efficiency. Empirical results on transport, reaction, and reaction-diffusion PDEs show 1–3 orders of magnitude improvements in accuracy over penalty and augmented-Lagrangian approaches, with demonstrated robustness to problem coefficients and tuning parameters, suggesting stronger, more reliable PDE solvers for SciML and potential extensions to inverse problems and operator learning.

Abstract

Physics-Informed Neural Networks (PINNs) represent a significant advancement in Scientific Machine Learning (SciML), which integrate physical domain knowledge into an empirical loss function as soft constraints and apply existing machine learning methods to train the model. However, recent research has noted that PINNs may fail to learn relatively complex Partial Differential Equations (PDEs). This paper addresses the failure modes of PINNs by introducing a novel, hard-constrained deep learning method -- trust-region Sequential Quadratic Programming (trSQP-PINN). In contrast to directly training the penalized soft-constrained loss as in PINNs, our method performs a linear-quadratic approximation of the hard-constrained loss, while leveraging the soft-constrained loss to adaptively adjust the trust-region radius. We only trust our model approximations and make updates within the trust region, and such an updating manner can overcome the ill-conditioning issue of PINNs. We also address the computational bottleneck of second-order SQP methods by employing quasi-Newton updates for second-order information, and importantly, we introduce a simple pretraining step to further enhance training efficiency of our method. We demonstrate the effectiveness of trSQP-PINN through extensive experiments. Compared to existing hard-constrained methods for PINNs, such as penalty methods and augmented Lagrangian methods, trSQP-PINN significantly improves the accuracy of the learned PDE solutions, achieving up to 1-3 orders of magnitude lower errors. Additionally, our pretraining step is generally effective for other hard-constrained methods, and experiments have shown the robustness of our method against both problem-specific parameters and algorithm tuning parameters.

Figures (9)

• Figure 1: Absolute and relative errors of three hard-constrained methods for learning PDEs.All three methods can learn PDE solutions well when the PDE coefficients $|\beta|$ and $|\alpha|$ are small, with trSQP-PINN yielding the lowest errors in solving transport and reaction-diffusion equations. However, as the coefficients increase and the problems become more challenging, trSQP-PINN significantly outperforms the other methods.
• Figure 2: Solutions of three hard-constrained methods for learning PDEs.TrSQP-PINN can fully recover PDE solutions at high coefficients, including initial conditions, periodic boundary conditions, and sharp transition patterns. The other methods fail to capture the complete solution features.
• Figure 3: Absolute and relative errors of three hard-constrained methods with varying number of training data points.TrSQP-PINN consistently outperforms the other methods and maintains low errors even if it is trained on a small dataset.
• Figure 4: PINN solutions for learning transport equation.PINN cannot recover the solution of the transport equation regardless of the penalty coefficient $\mu$ (see Figure \ref{['fig:2a']} for exact solution) and the prediction gets worse as $\mu$ increases.
• Figure 5: Solutions of penalty and augmented Lagrangian methods with/without pretraining for learning reaction and reaction-diffusion equations.For both methods, the predictions without pretraining (the left panels) are significantly worse than the predictions with pretraining (the right panels).

Theorems & Definitions (1)

• Remark 1