Improving the accuracy of physics-informed neural networks via last-layer retraining

TL;DR

A method for improving the accuracy of physics-informed neural networks by coupling them with a post-processing step that seeks the best approximation in a function space associated with the network, which yields errors four to five orders of magnitude lower than those of the parent PINNs.

Abstract

Physics-informed neural networks (PINNs) are a versatile tool in the burgeoning field of scientific machine learning for solving partial differential equations (PDEs). However, determining suitable training strategies for them is not obvious, with the result that they typically yield moderately accurate solutions. In this article, we propose a method for improving the accuracy of PINNs by coupling them with a post-processing step that seeks the best approximation in a function space associated with the network. We find that our method yields errors four to five orders of magnitude lower than those of the parent PINNs across architectures and dimensions. Moreover, we can reuse the basis functions for the linear space in more complex settings, such as time-dependent and nonlinear problems, allowing for transfer learning. Out approach also provides a residual-based metric that allows us to optimally choose the number of basis functions employed.

Figures (8)

• Figure 1: Poisson equation on the one dimensional domain $\Omega_1$. The errors $\lVert u_r - u\rVert$ for different values of $r$, measured in $L^{\infty}$ (blue) and $L^2$ (red), for problem (i), across PINN architectures. We also show the PINN errors $\lVert u^{\theta}_\text{NN} - u\rVert$ in $L^{\infty}$ (yellow) and $L^2$ (purple) as the horizontal curves. The residuals $\lVert e_r\rVert$ in in $L^{\infty}$ (green) and $L^2$ (cyan) are also shown; note the close similarity in the behaviour of the error and residual curves.
• Figure 2: Poisson equation in the square box $\Omega_2$. The errors $\lVert u_r - u\rVert$ for different values of $r$, measured in $L^{\infty}$ (blue) and $L^2$ (red), for problem (ii), across PINN architectures. We also show the PINN errors $\lVert u^{\theta}_\text{NN} - u\rVert$ in $L^{\infty}$ (yellow) and $L^2$ (purple) as the horizontal curves. The residuals $\lVert e_r\rVert$ in in $L^{\infty}$ (green) and $L^2$ (cyan) are also shown; note the close similarity in the behaviour of the error and residual curves.
• Figure 3: Poisson equation on the L-shaped domain $\Omega_3$. The errors $\lVert u_r - u\rVert$ for different values of $r$, measured in $L^{\infty}$ (blue) and $L^2$ (red), for problem (iii), across PINN architectures. We also show the PINN errors $\lVert u^{\theta}_\text{NN} - u\rVert$ in $L^{\infty}$ (yellow) and $L^2$ (purple) as the horizontal curves. The residuals $\lVert e_r\rVert$ in in $L^{\infty}$ (green) and $L^2$ (cyan) are also shown; note the close similarity in the behaviour of the error and residual curves.
• Figure 4: The heat equation on $\Omega_1 = [-1,1]$; we show the errors $E_{r,p}$ for different values of $r$, measured with $p = \infty$ (blue) and $p = 2$ (red). We also show present the $L^\infty$ (yellow) and $L^2$ (purple) norms of the residuals.
• Figure 5: The heat equation on the square box $\Omega_2$; the errors $E_{r,p}$ are shown for different values of $r$, measured with $p = \infty$ (blue) and $p = 2$ (red), as well as the averaged $L^\infty$ (yellow) and $L^2$ (purple) norms of the residuals.