Riemannian Laplace Approximation with the Fisher Metric

TL;DR

This work develops a principled Riemannian extension of Laplace approximation for Bayesian inference by leveraging the Fisher information metric as intrinsic geometry. It identifies biases in the prior RLA variant and introduces two improvements: a log-map correction (RLA-BLog) and a Fisher-metric based approach (RLA-F) that achieves asymptotic exactness for Gaussian-like targets and enhances finite-data performance. Theoretical results show conditions under which RLA-F is exact and Hausdorff MAP provides reparameterization-invariant optimization for Gaussian-like targets. Empirically, RLA-F consistently outperforms Euclidean LA and RLA-B across Banana, logistic regression, and neural-network tasks, while maintaining computational feasibility; RLA-B, though scalable, exhibits bias, especially in high dimensions. Overall, the Fisher-midelity RLA variant offers a robust, scalable, and theoretically grounded framework for accurate, efficient posterior approximations in complex models.

Abstract

Laplace's method approximates a target density with a Gaussian distribution at its mode. It is computationally efficient and asymptotically exact for Bayesian inference due to the Bernstein-von Mises theorem, but for complex targets and finite-data posteriors it is often too crude an approximation. A recent generalization of the Laplace Approximation transforms the Gaussian approximation according to a chosen Riemannian geometry providing a richer approximation family, while still retaining computational efficiency. However, as shown here, its properties depend heavily on the chosen metric, indeed the metric adopted in previous work results in approximations that are overly narrow as well as being biased even at the limit of infinite data. We correct this shortcoming by developing the approximation family further, deriving two alternative variants that are exact at the limit of infinite data, extending the theoretical analysis of the method, and demonstrating practical improvements in a range of experiments.

Figures (15)

• Figure 1: Left: The Fisher metric (green) captures the local curvature of the target density (black). Right: Samples from a Gaussian distribution (orange) are deterministically mapped using the Fisher metric to provide a flexible approximation (blue).
• Figure 2: Euclidean LA (left) is exact for a Gaussian target, but RLA-B (right) is biased. Lines are the true contours and marginals, with samples and histograms characterizing the approximation.
• Figure 3: Wasserstein distances from approximate samples to true samples for isotropic Gaussians of varying $D$, computed for the first dimension. The lines are means and $\text{means} \pm 2.0$ times $\text{standard deviations (stds)}$ computed over $5$ runs.
• Figure 4: RLA-F (left) is exact (already for finite data) for the squiggle distribution with complex shape due to diffeomorphism with a Gaussian, whereas RLA-B (right) is too narrow everywhere and generally biased.
• Figure 5: Left: The banana distribution has two Euclidean MAPs (blue) but only one Hausdorff MAP (brown). RLA-F depends on the MAP choice, with Hausdorff (middle) being here superior to Euclidean (right).

Theorems & Definitions (18)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Theorem 4