Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

TL;DR

This paper tackles uncertainty quantification in physics-informed neural networks (PINNs) for inverse problems by introducing RE-PINN, a repulsive ensemble method that adds a repulsive loss term to promote diversity among ensemble members. Building on Wasserstein gradient flow concepts, it derives a repulsive term in function space and extends it to the joint space of function values $f$ and differential-equation parameters $\boldsymbol{\lambda}$, approximating the ensemble density $\rho$ with kernel density estimation (KDE). The approach yields ensemble posteriors that converge toward the true Bayesian posterior $p(f,\lambda|\mathcal{D})$ in the limit of large ensembles and can be evaluated against Monte Carlo baselines. Across three PDE/inverse problems (exponential equation, damped harmonic oscillator, and advection equation), RE-PINN substantially improves uncertainty estimates over standard ensembles, with density-factorization strategies offering robust and accurate posterior representations in practice.

Abstract

Physics-informed neural networks (PINNs) have proven an effective tool for solving differential equations, in particular when considering non-standard or ill-posed settings. When inferring solutions and parameters of the differential equation from data, uncertainty estimates are preferable to point estimates, as they give an idea about the accuracy of the solution. In this work, we consider the inverse problem and employ repulsive ensembles of PINNs (RE-PINN) for obtaining such estimates. The repulsion is implemented by adding a particular repulsive term to the loss function, which has the property that the ensemble predictions correspond to the true Bayesian posterior in the limit of infinite ensemble members. Where possible, we compare the ensemble predictions to Monte Carlo baselines. Whereas the standard ensemble tends to collapse to maximum-a-posteriori solutions, the repulsive ensemble produces significantly more accurate uncertainty estimates and exhibits higher sample diversity.

Figures (11)

• Figure 1: Distribution of the repulsive ensemble when solving the exponential equation. The ensemble median is depicted, together with the 0.1-0.9 quantiles, and the 0.25-0.75 quantiles (shaded areas). The non-repulsive ensemble collapses to the MAP estimate. The dashed lines correspond to the MC baseline. The black dots depict training data. The ensembles have 50 members.
• Figure 2: Solving the exponential differential equation \ref{['eq:exp']} with repulsion in ($f$, $\lambda$)-space. Top row: The predictions in $f$-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9- quantiles as obtained via MCMC, together with the median. Black dots correspond to train data and red dots to test data. Bottom row: The predictions in $\lambda$-space are depicted. The ensemble distributions are compared to the distribution obtained via MCMC. The ensembles have 50 members.
• Figure 3: The evaluation metrics when solving the exponential equation with repulsion in ($f$, $\lambda$)-space with different numbers of ensemble members. For better visibility, the non-repulsive ensemble has been omitted for logL/N (test), since the values are far below those of the repulsive ensembles.
• Figure 4: Solving the damped harmonic oscillator \ref{['eq:dHO']} with repulsion in ($f$, $\lambda$)-space, using the repulsive ensemble with fully-factorized density. Left: The predictions in $f$-space are depicted. The light- and dark-shaded areas give the regions between the [0.1, 0.9]- and the [0.25, 0.75]-quantiles, respectively. The dashed lines depict the 0.1- and 0.9- quantiles as obtained via MCMC, together with the median. The non-repulsive ensemble effectively collapses to the MAP estimate. Black dots correspond to train data and red dots to test data. Middle: The predictions in $\lambda$-space are depicted. The ensemble distribution is compared to the distribution obtained via MCMC. Right: The individual ensemble members are depicted. The ensembles have 25 members.
• Figure 5: Example solution of the advection equation. Black dots correspond to training data and red dots to test data.