Extremization to Fine Tune Physics Informed Neural Networks for Solving Boundary Value Problems

TL;DR

This work tackles the training inefficiency of physics-informed neural networks (PINNs) for boundary value problems by blending deep neural networks with Gauss-Newton Extremization (GNE) and enforcing initial/boundary conditions exactly through Theory of Functional Connections (TFC) and Reduced TFC. The approach leverages the representational power of deep nets while enabling rapid fine-tuning of the final layer via GNE, significantly accelerating convergence and improving accuracy across ODEs, PDEs, and coupled PDEs, including stiff systems and Navier–Stokes-related problems such as Kovasznay flow and Taylor-Green vortices. Results show substantial RMSR reductions and speedups (up to tens of times) relative to standard PINN and ELM baselines, with Reduced TFC delivering dramatic training-time improvements and extending applicability to more complex boundary geometries. The method demonstrates strong potential for practical PINN deployment, though it acknowledges sensitivity to large gradients and suggests future enhancements (multi-GPU GNE, analytic Jacobians, and non-gradient optimizers).

Abstract

We propose a novel method for fast and accurate training of physics-informed neural networks (PINNs) to find solutions to boundary value problems (BVPs) and initial boundary value problems (IBVPs). By combining the methods of training deep neural networks (DNNs) and Extreme Learning Machines (ELMs), we develop a model which has the expressivity of DNNs with the fine-tuning ability of ELMs. We showcase the superiority of our proposed method by solving several BVPs and IBVPs which include linear and non-linear ordinary differential equations (ODEs), partial differential equations (PDEs) and coupled PDEs. The examples we consider include a stiff coupled ODE system where traditional numerical methods fail, a 3+1D non-linear PDE, Kovasznay flow and Taylor-Green vortex solutions to incompressible Navier-Stokes equations and pure advection solution of 1+1 D compressible Euler equation. The Theory of Functional Connections (TFC) is used to exactly impose initial and boundary conditions (IBCs) of (I)BVPs on PINNs. We propose a modification to the TFC framework named Reduced TFC and show a significant improvement in the training and inference time of PINNs compared to IBCs imposed using TFC. Furthermore, Reduced TFC is shown to be able to generalize to more complex boundary geometries which is not possible with TFC. We also introduce a method of applying boundary conditions at infinity for BVPs and numerically solve the pure advection in 1+1 D Euler equations using these boundary conditions.

Figures (27)

• Figure 1: Structure of a fully connect neural network solving a BVP with $k$ coupled differential equations $\mathcal{D}_1, \cdots, \mathcal{D}_k$ and $n$ initial and boundary conditions $\mathcal{B}_1, \cdots, \mathcal{B}_n$ acting on $l$ functions dependent on $m$ variables $x_1, \cdots, x_m$
• Figure 2: Root mean square of the residual of the ODE for neural networks with (left) 1 hidden layer with 400 neurons and (right) 4 hidden layers with (32, 32, 32, 400) neurons, trained on the interval $t \in [0, t_{max}]$.
• Figure 3: Comparison of the exact solution with the neural network approximation computed using a different training method on the domain $t \in [0,100]$. The neural network had 4 hidden layers with (32, 32, 32, 400) neurons. Note that the error in the Adam solution is apparent when we zoom into a small section of the domain (right panel).
• Figure 4: The absolute error in the solution after training a 4 hidden layer neural network with (32, 32, 32, 400) neurons for 100,000 steps on the domain $t \in [0,100]$.
• Figure 5: Learning curve of a DNN with 4 hidden layers with (32, 32, 32, 400) neurons trained on the domain $t \in [0,100]$.