Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecification

Terje Mildner, Paris Giampouras, Theodoros Damoulas

TL;DR

The paper addresses posterior inference under prior and model misspecification by introducing Generalised Variational Inference (GVI), which optimizes a loss term plus a bounded-divergence regularizer over a restricted space of posteriors. It establishes existence and (under mild conditions) uniqueness of GVI posteriors on arbitrary Polish spaces, proves concentration around loss minimisers, and derives rates of convergence just slower than 1/n that hold uniformly over priors when divergences are bounded. The results demonstrate robustness to severe prior misspecification, provide finite-sample and generalisation bounds, and extend to (partial) consistency results for unbounded divergences under additional assumptions. The framework has practical implications for scalable, robust posterior inference, including Federated GVI, where local private data can be integrated without being overly swayed by misspecified priors or models.

Abstract

We prove rates of convergence and robustness to prior misspecification within a Generalised Variational Inference (GVI) framework with bounded divergences. This addresses a significant open challenge for GVI and Federated GVI that employ a different divergence to the Kullback-Leibler under prior misspecification, operate within a subset of possible probability measures, and result in intractable posteriors. Our theoretical contributions extend to misspecified priors that lead to inconsistent Bayes posteriors. In particular, we are able to establish sufficient conditions for existence and uniqueness of GVI posteriors on arbitrary Polish spaces, prove that the GVI posterior measure concentrates on a neighbourhood of loss minimisers, and extend this to rates of convergence regardless of the prior measure.

Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecification

TL;DR

The paper addresses posterior inference under prior and model misspecification by introducing Generalised Variational Inference (GVI), which optimizes a loss term plus a bounded-divergence regularizer over a restricted space of posteriors. It establishes existence and (under mild conditions) uniqueness of GVI posteriors on arbitrary Polish spaces, proves concentration around loss minimisers, and derives rates of convergence just slower than 1/n that hold uniformly over priors when divergences are bounded. The results demonstrate robustness to severe prior misspecification, provide finite-sample and generalisation bounds, and extend to (partial) consistency results for unbounded divergences under additional assumptions. The framework has practical implications for scalable, robust posterior inference, including Federated GVI, where local private data can be integrated without being overly swayed by misspecified priors or models.

Abstract

We prove rates of convergence and robustness to prior misspecification within a Generalised Variational Inference (GVI) framework with bounded divergences. This addresses a significant open challenge for GVI and Federated GVI that employ a different divergence to the Kullback-Leibler under prior misspecification, operate within a subset of possible probability measures, and result in intractable posteriors. Our theoretical contributions extend to misspecified priors that lead to inconsistent Bayes posteriors. In particular, we are able to establish sufficient conditions for existence and uniqueness of GVI posteriors on arbitrary Polish spaces, prove that the GVI posterior measure concentrates on a neighbourhood of loss minimisers, and extend this to rates of convergence regardless of the prior measure.

Paper Structure

This paper contains 26 sections, 14 theorems, 55 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation
  4. Assumptions
  5. Assumptions
  6. Misspecification of the Model and the Prior
  7. Generalised Variational Inference
  8. Main Results
  9. Finite Sample Bound and Extensions
  10. A Finite Sample Bound
  11. Generalisation Performance
  12. On Consistency of GVI Posteriors under Arbitrary Divergences
  13. Illustrative Example
  14. Mitigating Inconsistent Bayes Posteriors
  15. The Choice of Divergence
  16. ...and 11 more sections

Key Result

Theorem 1

Make asp:lossasp:lscLasp:lscDasp:finite. Suppose also that $\mathcal{Q}$ is closed with respect to weak$^\star$ convergence. Then a GVI posterior minimising eqn:gvi_min exists. If it further holds that the divergence is strictly convex in its first argument and that $\mathcal{Q}$ is convex, then the

Figures (1)

  • Figure 1: A taxonomy of Generalised Variational Inference.

Theorems & Definitions (37)

  • Remark
  • Definition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Remark
  • Definition 2
  • Definition 2
  • Proposition 5
  • ...and 27 more