Bayesian Experimental Design for Model Discrepancy Calibration: A Rivalry between Kullback--Leibler Divergence and Wasserstein Distance

TL;DR

This paper investigates how choice of utility in Bayesian experimental design (BED) influences sequential data collection, focusing on KL divergence and Wasserstein distance as criteria. It demonstrates a location-dependent false reward for Wasserstein in simple and PDE-based inverse problems, showing KL often yields faster convergence without model discrepancy while Wasserstein provides more robust updates when model discrepancy is present. The work further introduces a discrepancy-correction framework using AD-EKI that separates low-dimensional physical parameters from high-dimensional neural discrepancy parameters and employs a re-update strategy to leverage accumulated data. Across toy and convection-diffusion examples, the results delineate trade-offs: KL is typically more efficient with accurate forward models, whereas Wasserstein offers robustness and faster correction under misspecification, with the gap narrowing when using expected utilities. The findings provide practical guidelines on selecting BED utility functions based on the likelihood of model discrepancy and the reliance on expected versus actual information gains.

Abstract

Designing experiments that systematically gather data from complex physical systems is central to accelerating scientific discovery. While Bayesian experimental design (BED) provides a principled, information-based framework that integrates experimental planning with probabilistic inference, the selection of utility functions in BED is a long-standing and active topic, where different criteria emphasize different notions of information. Although Kullback--Leibler (KL) divergence has been one of the most common choices, recent studies have proposed Wasserstein distance as an alternative. In this work, we first employ a toy example to illustrate an issue of Wasserstein distance - the value of Wasserstein distance of a fixed-shape posterior depends on the relative position of its main mass within the support and can exhibit false rewards unrelated to information gain, especially with a non-informative prior (e.g., uniform distribution). We then further provide a systematic comparison between these two criteria through a classical source inversion problem in the BED literature, revealing that the KL divergence tends to lead to faster convergence in the absence of model discrepancy, while Wasserstein metrics provide more robust sequential BED results if model discrepancy is non-negligible. These findings clarify the trade-offs between KL divergence and Wasserstein metrics for the utility function and provide guidelines for selecting suitable criteria in practical BED applications.

Figures (13)

• Figure 1: Illustration of design and associated reward. Panel (a) Design and posterior: each filled circle denotes a design point, and the dotted circle of the same color indicates its corresponding posterior distribution. The red star marks the true source location. Panel (b) Reward vs. design: Wasserstein-1 and Wasserstein-2 distances are plotted as functions of the design location, normalized so that $0$ corresponds to the source location and $1$ to the domain corner. The grey dotted line shows the baseline.
• Figure 2: Without error (case 1), using actual information gain: posterior. From top to bottom: KL, $W_2$, $W_1$. The parameter space is set as $[0,1]^2$ but zoomed in as $[0,0.6]^2$. This holds for all the remaining posterior figures.
• Figure 3: Distance and uncertainty evolution.
• Figure 4: Reward map with no model discrepancy at stage 1. The black dotted lines indicate the $z_x$ and $z_y$ coordinates of the global maximizer.
• Figure 5: Without error (case 2), using expected information gain: posterior. From top to bottom: KL, $W_2$, $W_1$.