Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Bayesian Optimal Experimental Design with Normalizing Flows

Jiayuan Dong, Christian Jacobsen, Mehdi Khalloufi, Maryam Akram, Wanjiao Liu, Karthik Duraisamy, Xun Huan

TL;DR

The paper addresses efficient Bayesian optimal experimental design when likelihoods are implicit or expensive, by replacing exact posterior computations with a variational bound based on the Barber–Agakov construction. It introduces vOED-NFs, which use conditional invertible neural networks (NFs) to flexibly approximate posteriors and to compute a tight lower bound on the expected information gain, enabling gradient-based design optimization under a fixed forward-model budget. Across four case studies, vOED-NFs deliver tighter bounds and posterior approximations that capture non-Gaussian and multi-modal features, confirming superior information gain estimation relative to Gaussian variational baselines and some NMC variants. The results highlight the method’s potential for PDE-governed experiments and stochastic models with implicit likelihoods, while also outlining limitations and avenues for scalable transport-map architectures and broader OED extensions. Collectively, the work advances likelihood-free, NF-based variational design methods with practical implications for efficient experimental planning in complex physical systems.

Abstract

Bayesian optimal experimental design (OED) seeks experiments that maximize the expected information gain (EIG) in model parameters. Directly estimating the EIG using nested Monte Carlo is computationally expensive and requires an explicit likelihood. Variational OED (vOED), in contrast, estimates a lower bound of the EIG without likelihood evaluations by approximating the posterior distributions with variational forms, and then tightens the bound by optimizing its variational parameters. We introduce the use of normalizing flows (NFs) for representing variational distributions in vOED; we call this approach vOED-NFs. Specifically, we adopt NFs with a conditional invertible neural network architecture built from compositions of coupling layers, and enhanced with a summary network for data dimension reduction. We present Monte Carlo estimators to the lower bound along with gradient expressions to enable a gradient-based simultaneous optimization of the variational parameters and the design variables. The vOED-NFs algorithm is then validated in two benchmark problems, and demonstrated on a partial differential equation-governed application of cathodic electrophoretic deposition and an implicit likelihood case with stochastic modeling of aphid population. The findings suggest that a composition of 4--5 coupling layers is able to achieve lower EIG estimation bias, under a fixed budget of forward model runs, compared to previous approaches. The resulting NFs produce approximate posteriors that agree well with the true posteriors, able to capture non-Gaussian and multi-modal features effectively.

Variational Bayesian Optimal Experimental Design with Normalizing Flows

TL;DR

The paper addresses efficient Bayesian optimal experimental design when likelihoods are implicit or expensive, by replacing exact posterior computations with a variational bound based on the Barber–Agakov construction. It introduces vOED-NFs, which use conditional invertible neural networks (NFs) to flexibly approximate posteriors and to compute a tight lower bound on the expected information gain, enabling gradient-based design optimization under a fixed forward-model budget. Across four case studies, vOED-NFs deliver tighter bounds and posterior approximations that capture non-Gaussian and multi-modal features, confirming superior information gain estimation relative to Gaussian variational baselines and some NMC variants. The results highlight the method’s potential for PDE-governed experiments and stochastic models with implicit likelihoods, while also outlining limitations and avenues for scalable transport-map architectures and broader OED extensions. Collectively, the work advances likelihood-free, NF-based variational design methods with practical implications for efficient experimental planning in complex physical systems.

Abstract

Bayesian optimal experimental design (OED) seeks experiments that maximize the expected information gain (EIG) in model parameters. Directly estimating the EIG using nested Monte Carlo is computationally expensive and requires an explicit likelihood. Variational OED (vOED), in contrast, estimates a lower bound of the EIG without likelihood evaluations by approximating the posterior distributions with variational forms, and then tightens the bound by optimizing its variational parameters. We introduce the use of normalizing flows (NFs) for representing variational distributions in vOED; we call this approach vOED-NFs. Specifically, we adopt NFs with a conditional invertible neural network architecture built from compositions of coupling layers, and enhanced with a summary network for data dimension reduction. We present Monte Carlo estimators to the lower bound along with gradient expressions to enable a gradient-based simultaneous optimization of the variational parameters and the design variables. The vOED-NFs algorithm is then validated in two benchmark problems, and demonstrated on a partial differential equation-governed application of cathodic electrophoretic deposition and an implicit likelihood case with stochastic modeling of aphid population. The findings suggest that a composition of 4--5 coupling layers is able to achieve lower EIG estimation bias, under a fixed budget of forward model runs, compared to previous approaches. The resulting NFs produce approximate posteriors that agree well with the true posteriors, able to capture non-Gaussian and multi-modal features effectively.
Paper Structure (27 sections, 1 theorem, 42 equations, 15 figures, 11 tables)

This paper contains 27 sections, 1 theorem, 42 equations, 15 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Bayesian optimal experimental design
  4. Variational optimal experimental design
  5. Computational Methods
  6. Normalizing flows
  7. Numerical Experiments
  8. Case 1: Low-dimensional nonlinear design
  9. Case 2: High-dimensional linear design
  10. Case 3: Cathodic electrophoretic deposition (e-coating)
  11. Case 4: aphid population
  12. Conclusions
  13. Proof for \ref{['prop:ordering']}
  14. Additional Discussions for Invertible Neural Networks
  15. Case 1
  16. ...and 12 more sections

Key Result

Proposition 1

Consider probability densities $q(\boldsymbol{\theta}|\textbf{y};\boldsymbol{\lambda}_1)$ and $q(\boldsymbol{\theta}|\textbf{y};\boldsymbol{\lambda}_2)$ formed at variational parameter values $\boldsymbol{\lambda}_1$ and $\boldsymbol{\lambda}_2$, respectively, both as approximations to the true post if and only if

Figures (15)

  • Figure 1: Diagram for the cINN NFs transformation.
  • Figure 2: Case 1. EIG estimates and sample training convergence plot. The Re-NMC plot line is right on top of the High-NMC plot line, as they produce near-identical results.
  • Figure 3: Case 1. Comparison of marginal posteriors obtained from SMC and vOED-NFs at $d = 0.2$.
  • Figure 4: Case 1. Comparison of marginal posteriors obtained from SMC and vOED-NFs at $d = 1.0$.
  • Figure 5: Case 1. EIG under different number of transformations $T$ and optimization sample sizes $N_{\text{opt}}$.
  • ...and 10 more figures

Theorems & Definitions (2)

  • Proposition 1
  • proof