Bayesian Reasoning for Physics Informed Neural Networks

TL;DR

This work integrates physics-informed neural networks with a Bayesian framework using the Laplace approximation to obtain weight posteriors and model evidence for model selection. By decomposing the PINN loss into equation and boundary contributions and treating hyperparameters as inferable quantities, the method automatically tunes loss weights and provides quantified uncertainties $p(y|\cdot)$ through the posterior and the evidence. The authors demonstrate the approach on the heat, wave, and Burger’s equations, showing that the Bayesian PINN achieves accurate predictions with credible uncertainty intervals and competitive evidence compared to HMC-based methods. The strategy enables epistemic uncertainty quantification and principled model selection in PINNs for PDE problems, particularly for shallow networks where the Gaussian approximation and Hessian-based computations are tractable.

Abstract

We present the application of the physics-informed neural network (PINN) approach in Bayesian formulation. We have adopted the Bayesian neural network framework to obtain posterior densities from Laplace approximation. For each model or fit, the evidence is computed, which is a measure that classifies the hypothesis. The optimal solution is the one with the highest value of evidence. We have proposed a modification of the Bayesian algorithm to obtain hyperparameters of the model. We have shown that within the Bayesian framework, one can obtain the relative weights between the boundary and equation contributions to the total loss. Presented method leads to predictions comparable to those obtained by sampling from the posterior distribution within the Hybrid Monte Carlo algorithm (HMC). We have solved heat, wave, and Burger's equations, and the results obtained are in agreement with the exact solutions, demonstrating the effectiveness of our approach. In Burger's equation problem, we have demonstrated that the framework can combine information from differential equations and potential measurements. All solutions are provided with uncertainties (induced by the model's parameter dependence) computed within the Bayesian framework.

Figures (12)

• Figure 1: The above MLP contains two hidden unit layers. Empty squares denote the input, whereas filled circles represent the output; Blue filled and open circles indicate hidden and bias units, respectively.
• Figure 2: Nonlinear regression within this paper approach (left panel) and HMC approach (right panel). The red/blue line corresponds to this paper/HMC fit. The solid blue line describes the true value.
• Figure 3: In the left panel: the training data, the blue/red points correspond to the equation/boundary contribution to the loss for the heat equation. In the right panel: the histogram of the relative log of evidence (the difference between a given log of evidence and the highest log of evidence).
• Figure 4: An example of loss $E_T$ (the first panel in the left), $E_{hyp}$ (the last panel) and $\alpha$ (the second from the right) and $\beta$ (the third from the right) hyperparameters evolution during the training of the network that solves the heat equation.
• Figure 5: The exact ($u_{true}$) and PINN ($u_{pred}$) solutions for the heat equation. The grey area denotes $2\sigma$ uncertainty.