Latent-IMH: Efficient Bayesian Inference for Inverse Problems with Approximate Operators

TL;DR

Latent-IMH addresses Bayesian inverse problems with expensive forward operators by introducing a latent variable approach driven by an offline-constructed approximate operator. It builds a cheap proposal using the approximate operator, then refines samples via the exact forward model within an Independence Metropolis-Hastings framework, yielding a posterior consistent with the exact model. The authors derive KL-divergence bounds and mixing-time guarantees, and demonstrate through diverse numerical tests that Latent-IMH achieves higher efficiency than NUTS and MALA, including in multimodal settings where conventional samplers struggle. The approach shifts substantial computational work to offline precomputation, enabling scalable Bayesian inference for large-scale inverse problems, while acknowledging limitations related to memory, linearity, and the dependence on a good offline approximation.

Abstract

We study sampling from posterior distributions in Bayesian linear inverse problems where $A$, the parameters to observables operator, is computationally expensive. In many applications, $A$ can be factored in a manner that facilitates the construction of a cost-effective approximation $\tilde{A}$. In this framework, we introduce Latent-IMH, a sampling method based on the Metropolis-Hastings independence (IMH) sampler. Latent-IMH first generates intermediate latent variables using the approximate $\tilde{A}$, and then refines them using the exact $A$. Its primary benefit is that it shifts the computational cost to an offline phase. We theoretically analyze the performance of Latent-IMH using KL divergence and mixing time bounds. Using numerical experiments on several model problems, we show that, under reasonable assumptions, it outperforms state-of-the-art methods such as the No-U-Turn sampler (NUTS) in computational efficiency. In some cases, Latent-IMH can be orders of magnitude faster than existing schemes.

Figures (10)

• Figure 1: Reconstructed acoustic source fields. We reconstruct the sound source field $x$ from partial observations under a non-Gaussian prior, where posterior inference is only possible via sampling. From left to right: $x_\text{true}$ is the ground truth, and $x_\text{mean}$ is an accurate posterior mean computed from thousands of samples. The Approx-IMH and Latent-IMH rows show posterior averages after 200, 500, 2000, and 5000 MCMC steps. The last column shows the relative mean error and squared bias of the second moment between the true posterior and sampled estimates as a function of the number of forward ${\bf F}$ and inverse ${\bf F}^{-1}$ solves. (MALA: Metropolis Adjusted Langevin sampler; NUTS: No-U-Turn Sampler.) Latent-IMH achieves high accuracy with substantial computational savings: for 10% relative mean error, it requires $\sim 10^3$ solves, Approx-IMH $\sim 10^5$, and MALA/NUTS millions of solves.
• Figure 2: Sensitivity test results for the expected KL-divergence of Approx-IMH posterior and Latent-IMH posterior relative to Exact posterior.
• Figure 3: Acceptance ratio of Approx-IMH and Latent-IMH for different problem scenarios.
• Figure 4: Sample efficiency comparison for Gaussian priors: results averaged over 5 independent runs. Left (Gaussian mixture prior): Numbers in parentheses in the top legend indicate spectral error $\|{\bf I} - \widetilde{{\bf F}}^{-1} {\bf F}\|_2$; numbers above histogram plots denote the total number of forward and inverse solves of ${\bf F}$ required by each sampler. Right (standard normal prior): The approximate operator in the mean error plot has spectral error of 4.7%.
• Figure 5: Results for Laplace prior test with normalizing flow. All results averaged over 5 independent runs. Numbers in parentheses indicate spectral error $\|{\bf I} - \widetilde{{\bf F}}^{-1} {\bf F}\|_2$.

Theorems & Definitions (20)

• Proposition 3.2
• Proposition 3.3
• Theorem 4.3
• Theorem 4.4
• Proposition A.1: Construction of ${\bf V}_x$ and ${\bf Z}$ in \ref{['eq:transform']}
• proof
• Theorem B.2
• Lemma B.3: KL-divergence between two multivariate Gaussians
• proof : Proof of \ref{['prop:kl-closedform']}
• proof