Neural Networks for Bayesian Inverse Problems Governed by a Nonlinear ODE

TL;DR

This work develops a neural-network framework to solve Bayesian inverse problems for nonlinear ODEs, specifically the FitzHugh–Nagumo model, enabling end-to-end estimation of dynamical parameters, noise parameters, and posterior covariance from noisy time-series data. By training NN surrogates on forward simulations and using a Laplace-based Hessian approach in a log-Euclidean space, the method delivers uncertainty quantification with a single forward pass, while carefully handling the ill-posedness of the inverse problem. The authors compare dense and convolutional architectures, find CNNs generally outperform DNNs for time-series data, and demonstrate scalability to higher-dimensional outputs and different noise models, including AR additive noise and SDE-based intrinsic noise, with substantial HPC speedups. The approach provides a practical, likelihood-free pathway to parameter inference and uncertainty propagation in dynamical systems, with potential applications in neuroscience and other domains involving noisy, nonlinear dynamics.

Abstract

We investigate the use of neural networks (NNs) for the estimation of hidden model parameters and uncertainty quantification from noisy observational data for inverse parameter estimation problems. We formulate the parameter estimation as a Bayesian inverse problem. We consider a parametrized system of nonlinear ordinary differential equations (ODEs), which is the FitzHugh--Nagumo model. The considered problem exhibits significant mathematical and computational challenges for classical parameter estimation methods, including strong nonlinearities, nonconvexity, and sharp gradients. We explore how NNs overcome these challenges by approximating reconstruction maps for parameter estimation from observational data. The considered data are time series of the spiking membrane potential of a biological neuron. We infer parameters controlling the dynamics of the model, noise parameters of autocorrelated additive noise, and noise modeled via stochastic differential equations, as well as the covariance matrix of the posterior distribution to expose parameter uncertainties--all with just one forward evaluation of an appropriate NN. We report results for different NN architectures and study the influence of noise on prediction accuracy. We also report timing results for training NNs on dedicated hardware. Our results demonstrate that NNs are a versatile tool to estimate parameters of the dynamical system, stochastic processes, as well as uncertainties, as they propagate through the governing ODE.

Figures (16)

• Figure 1: Illustration of the model dynamics and noise perturbations. We show time series of the membrane potential $u$ for $\tau = 200\,\text{ms}$. Left: Solution of the FitzHugh--Nagumo model (membrane potential $u$) for representative model parameters $\boldsymbol{\theta}_{\text{dyn}} \in \mathbb{R}^3$. Right: Plots of the observational data $y_{\text{obs}}$ for an additive and an intrinsic noise model. We limit the observational data associated with the FitzHugh--Nagumo model to the membrane potential $u$. Our goal is to learn a map, $g_{\Xi}$, from the observational data $y_{\text{obs}}$ to the hidden parameters, $\boldsymbol{\theta}_{\text{pred}} = g(y_{\text{obs}})$, such that $\boldsymbol{\theta}_{\text{pred}} \approx \boldsymbol{\theta}_{\text{true}}$.
• Figure 1: Illustration of the CNN architecture. The input is passed into a sequence of convolutional block (each consisting of a convolution, activation, and pooling layers); the output of the last convolutional block is passed to a sequence of dense blocks (each consisting of a dense and activation layers). The last dense layer, without activation, predicts parameters.
• Figure 1: Scatter plots for CNN predictions using testing data when estimating $\boldsymbol{\theta}_{\text{dyn}} = (\theta_0,\theta_1,\theta_2) \in \mathbb{R}^3$ for the data type TS&FC and a training size of $n_{\text{train}} = 8000.0$. The CNN also infers $\boldsymbol{\theta}_{\text{noise}}$, but this is not shown. The three vertical blocks correspond to additive noise (top block), intrinsic noise (middle block), and combined noise (bottom block). We plot the value of the test samples ($x$-axis) versus the predicted values ($y$-axis) for $\theta_0$ (left), $\theta_1$ (center), and $\theta_2$ (right). The diagonal red line indicates optimal predictions. Overall, the CNNs deliver similar predictions across different noise configurations.
• Figure 1: Singular values $s_i$, $i=1,2,3$, for the positive-definite Hessian matrices $\boldsymbol{H}$. From left to right we show the box-whisker plots for the singular values $s_1$, $s_2$, and $s_3$, respectively. The original dataset consists of 15000.0 samples. 110 of these matrices were negative definite in computation. Here, we show the singular values for the remaining 14890.0 matrices. Dashed horizontal lines correspond to the upper and lower whiskers of the plots.
• Figure 2: Illustration of the parameter estimation problem. We show a contour plot and the associated surface plot of the objective that enters our variational optimization problem as a function of the hidden parameters $\theta_0$ and $\theta_1$ of $\boldsymbol{\theta}_{\text{dyn}} = (\theta_0, \theta_1, \theta_2) \in \mathbb{R}^3$ that control the model output. We can observe that the optimization problem is nonconvex with a volatile change in objective values that features a narrow, flat optimality zone (dark blue region in plot on the left) with sharp gradients (surface plot on the right).

Theorems & Definitions (3)

• Remark 2.1
• Remark 2.2
• Remark 2.3