Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flows

TL;DR

This work addresses Bayesian inverse problems where the forward model is expensive, gradients may be unavailable, and the posterior is potentially multimodal. It develops GMKI, a derivative-free sampler that combines Fisher-Rao gradient flows, Gaussian mixture approximations, and Kalman-type updates, achieved via an operator-splitting time discretization into exploration and exploitation steps. Theoretical results establish exponential convergence in the continuous-time limit, affine invariance, and exploration-inspired dynamics, while numerical experiments demonstrate GMKI’s ability to recover multiple modes efficiently in 1D, 2D, and a high-dimensional Navier–Stokes initial-condition problem. The approach offers a practical, scalable framework for fast posterior approximation in challenging inverse problems where forward evaluations are costly and gradient information is unavailable. Overall, GMKI provides a principled, efficient, and scalable derivative-free method for multimodal Bayesian inference with broad applicability in science and engineering.

Abstract

In this paper, we study efficient approximate sampling for probability distributions known up to normalization constants. We specifically focus on a problem class arising in Bayesian inference for large-scale inverse problems in science and engineering applications. The computational challenges we address with the proposed methodology are: (i) the need for repeated evaluations of expensive forward models; (ii) the potential existence of multiple modes; and (iii) the fact that gradient of, or adjoint solver for, the forward model might not be feasible. While existing Bayesian inference methods meet some of these challenges individually, we propose a framework that tackles all three systematically. Our approach builds upon the Fisher-Rao gradient flow in probability space, yielding a dynamical system for probability densities that converges towards the target distribution at a uniform exponential rate. This rapid convergence is advantageous for the computational burden outlined in (i). We apply Gaussian mixture approximations with operator splitting techniques to simulate the flow numerically; the resulting approximation can capture multiple modes thus addressing (ii). Furthermore, we employ the Kalman methodology to facilitate a derivative-free update of these Gaussian components and their respective weights, addressing the issue in (iii). The proposed methodology results in an efficient derivative-free sampler flexible enough to handle multi-modal distributions: Gaussian Mixture Kalman Inversion (GMKI). The effectiveness of GMKI is demonstrated both theoretically and numerically in several experiments with multimodal target distributions, including proof-of-concept and two-dimensional examples, as well as a large-scale application: recovering the Navier-Stokes initial condition from solution data at positive times.

Figures (10)

• Figure 1: Schematic of properties of GMKI. The Grey curve represents the posterior distribution. Blue curves represent Gaussian components of the Gaussian mixture approximation. From left to right: Gaussian components can exhibit exponential convergence toward their respective Gaussian modes if these modes are well separated (see \ref{['lem:GMKI-simplified']}); the repulsion between distinct Gaussian components in the iteration of GMKI helps explore the space and capture multiple modes (see \ref{['ssec:exploration']}); when multiple Gaussian components converge towards a single Gaussian mode in the posterior distribution, they can provide a good approximation of the Gaussian mode (see \ref{['lem:GMKI-linear']}); GMKI can capture multiple modes even when these modes are intertwined (see numerical examples in \ref{['ssec:1d-bimodal']}).
• Figure 2: The one-dimensional bimodal problem with $\Sigma_{\eta}$ values of $0.2^2$ (top), $0.5^2$ (top middle), $1.0^2$ (bottom middle), and $1.5^2$ (bottom). Each panel displays the reference posterior distribution (grey square lines) and posterior distributions estimated by the GMKI (blue lines) at the $30$th iteration with mode number $K=1,\,2,\,3$ (from left to right) with mean $m_k$ (colored) and initial mean (black) of each Gaussian component marked. The fourth figure shows the total variation distance between the reference posterior distribution and the posterior distributions estimated by the GMKI with mode number $K=1,\,2,\,3$.
• Figure 3: Two-dimensional bimodal problem with $\rho_{\rm prior} \sim \mathcal{N}(0, I)$. From left to right: reference posterior distribution (left), posterior distributions estimated by 3-modal GMKI (middle left) at the 30th iteration (means $m_k$ (colored) and initial means (black) are marked), BDLS-KL lu2022birth (middle right) at the 1000th iteration, and total variation distance between the reference posterior distribution and the posterior distributions estimated by the GMKI (right).
• Figure 4: Two-dimensional bimodal problem with $\rho_{\rm prior} \sim \mathcal{N}([0.5, 0]^T, I)$. From left to right: reference posterior distribution (left), posterior distributions estimated by 3-modal GMKI (middle left) at the 30th iteration (means $m_k$ (colored) and initial means (black) are marked), BDLS-KL lu2022birth (middle right) at the 1000th iteration, and total variation distance between the reference posterior distribution and the posterior distributions estimated by the GMKI (right).
• Figure 5: The vorticity field $\omega$ at $T=0.25$ and $T=0.5$ and observations $\omega([x_{(1)}, x_{(2)}]^T) - \omega([2\pi - x_{(1)}, x_{(2)}]^T)$ at $56$ equidistant points (solid black dots). Their mirroring points are marked (empty black dots).

Theorems & Definitions (12)

• Proposition 1
• Proposition 2
• Remark 1
• Proposition 3
• Proposition 4
• Proposition 5
• Remark 2
• Proposition 6: Linear inverse problems
• Remark 3
• Proposition 7