Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Adaptive Calibration and Optimal Design

Rafael Oliveira, Dino Sejdinovic, David Howard, Edwin V. Bonilla

TL;DR

BACON tackles calibrating computer models under expensive simulations by treating design selection and calibration as a joint Bayesian adaptive design problem. It replaces intractable expected information gain with a variational lower bound, enabling simultaneous optimization of designs, calibration inputs, and a flexible posterior (including conditional normalising flows). The approach leverages a bi-fidelity Gaussian process to couple real observations and simulator outputs, achieving superior information gain and posterior accuracy compared to baselines, especially in multimodal settings. Scalability is addressed through sparse GP extensions and amortised inference ideas, broadening applicability to physics- and engineering-scale calibration tasks.

Abstract

The process of calibrating computer models of natural phenomena is essential for applications in the physical sciences, where plenty of domain knowledge can be embedded into simulations and then calibrated against real observations. Current machine learning approaches, however, mostly rely on rerunning simulations over a fixed set of designs available in the observed data, potentially neglecting informative correlations across the design space and requiring a large amount of simulations. Instead, we consider the calibration process from the perspective of Bayesian adaptive experimental design and propose a data-efficient algorithm to run maximally informative simulations within a batch-sequential process. At each round, the algorithm jointly estimates the parameters of the posterior distribution and optimal designs by maximising a variational lower bound of the expected information gain. The simulator is modelled as a sample from a Gaussian process, which allows us to correlate simulations and observed data with the unknown calibration parameters. We show the benefits of our method when compared to related approaches across synthetic and real-data problems.

Bayesian Adaptive Calibration and Optimal Design

TL;DR

BACON tackles calibrating computer models under expensive simulations by treating design selection and calibration as a joint Bayesian adaptive design problem. It replaces intractable expected information gain with a variational lower bound, enabling simultaneous optimization of designs, calibration inputs, and a flexible posterior (including conditional normalising flows). The approach leverages a bi-fidelity Gaussian process to couple real observations and simulator outputs, achieving superior information gain and posterior accuracy compared to baselines, especially in multimodal settings. Scalability is addressed through sparse GP extensions and amortised inference ideas, broadening applicability to physics- and engineering-scale calibration tasks.

Abstract

The process of calibrating computer models of natural phenomena is essential for applications in the physical sciences, where plenty of domain knowledge can be embedded into simulations and then calibrated against real observations. Current machine learning approaches, however, mostly rely on rerunning simulations over a fixed set of designs available in the observed data, potentially neglecting informative correlations across the design space and requiring a large amount of simulations. Instead, we consider the calibration process from the perspective of Bayesian adaptive experimental design and propose a data-efficient algorithm to run maximally informative simulations within a batch-sequential process. At each round, the algorithm jointly estimates the parameters of the posterior distribution and optimal designs by maximising a variational lower bound of the expected information gain. The simulator is modelled as a sample from a Gaussian process, which allows us to correlate simulations and observed data with the unknown calibration parameters. We show the benefits of our method when compared to related approaches across synthetic and real-data problems.
Paper Structure (44 sections, 25 equations, 4 figures, 3 tables, 4 algorithms)

This paper contains 44 sections, 25 equations, 4 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Related work
  4. Adaptive experimental design
  5. Active learning for calibration
  6. Gaussian processes for Bayesian calibration
  7. Bi-fidelity exact Gaussian process model
  8. Joint probabilistic model and predictions
  9. Bayesian adaptive calibration and optimal design
  10. Variational EIG lower bound
  11. Algorithm
  12. Variational posteriors
  13. Conditional Gaussian models.
  14. Conditional normalising flows
  15. Batch parallel evaluations
  16. ...and 29 more sections

Figures (4)

  • Figure 1: Experimental results on synthetic data where the target posterior $p^*$ is unimodal. The first 3 plots show estimates for performance metrics as a function of the number of simulations run (not including the initial data). Estimates were computed based on the posterior estimates for each method available during their run, with random using $p({\boldsymbol{\mathbf{\theta}}}^*)$, D-optimal and BACON using MCMC posteriors, and IMSPE using a Dirac delta (reverse KL undefined, not shown) on the MAP estimate as posterior estimates. Results are averaged over 10 trials, and shaded areas indicate $\pm 1$ standard deviation. The rightmost plot shows the target posterior, with the true ${\boldsymbol{\mathbf{\theta}}}^*$ indicated by a star.
  • Figure 2: Experimental results on synthetic data where the target posterior $p^*$ is bimodal. See \ref{['fig:synthetic-cal']} for details, with the exception that the rightmost plot now shows the bimodal target posterior.
  • Figure 3: Soft-robotics grasping experiment. We calibrate a soft materials simulator against real data from physical grasping from an automated experimentation platform
  • Figure 4: Final posterior approximations $p({\boldsymbol{\mathbf{\theta}}}^*|{\mathcal{D}}_T)$ and simulation parameter ${\boldsymbol{\mathbf{\hat{\theta}}}}$ (red crosses) choices by each method for the soft-robotics simulator calibration problem after one of the runs. The target/reference posterior (\ref{['fig:sofa-reference']}) was inferred using a large number (1024) of simulations following a Latin hypercube pattern over the combined design $\mathcal{X}$ and calibration parameters space $\Theta$ and a uniform prior $p({\boldsymbol{\mathbf{\theta}}})$ over the same range as the smooth uniform prior the algorithms used. The posteriors are plotted as a 2D histogram over the normalised range (after an affine and sigmoid transform), which the algorithms used for optimisation. The KL divergences in \ref{['tab:sofa-results']} are computed with respect to this reference posterior. Also note that the simulation parameters ${\boldsymbol{\mathbf{\hat{\theta}}}}$ in the plot correspond to different algorithmic choices for design inputs ${\boldsymbol{\mathbf{\hat{{\boldsymbol{\mathbf{x}}}}}}}$, which are 9-dimensional variables that are not plotted here.