Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Approximations for Robust Bayesian Inference via Rho-Posteriors

EL Mahdi Khribch, Pierre Alquier

TL;DR

The paper tackles robust Bayesian inference under contaminated data by developing a variational, PAC-Bayesian approach to the $\rho$-posterior. It introduces temperature-dependent Gibbs posteriors and derives finite-sample oracle inequalities that preserve robustness properties, including explicit contamination rates. To make the theory practical, it couples competitor-based risk with variational approximations and a saddle-point optimization framework, offering convergence guarantees in exponential-family settings. Localization via Catoni-style priors and structured variational schemes yield near-optimal rates and tractable computations. Extensive experiments across Gaussian, Poisson, Uniform models, regression with fixed design, and real datasets demonstrate strong robustness to contamination while maintaining competitive performance when well-specified.

Abstract

The $ρ$-posterior framework provides universal Bayesian estimation with explicit contamination rates and optimal convergence guarantees, but has remained computationally difficult due to an optimization over reference distributions that precludes intractable posterior computation. We develop a PAC-Bayesian framework that recovers these theoretical guarantees through temperature-dependent Gibbs posteriors, deriving finite-sample oracle inequalities with explicit rates and introducing tractable variational approximations that inherit the robustness properties of exact $ρ$-posteriors. Numerical experiments demonstrate that this approach achieves theoretical contamination rates while remaining computationally feasible, providing the first practical implementation of $ρ$-posterior inference with rigorous finite-sample guarantees.

Variational Approximations for Robust Bayesian Inference via Rho-Posteriors

TL;DR

The paper tackles robust Bayesian inference under contaminated data by developing a variational, PAC-Bayesian approach to the -posterior. It introduces temperature-dependent Gibbs posteriors and derives finite-sample oracle inequalities that preserve robustness properties, including explicit contamination rates. To make the theory practical, it couples competitor-based risk with variational approximations and a saddle-point optimization framework, offering convergence guarantees in exponential-family settings. Localization via Catoni-style priors and structured variational schemes yield near-optimal rates and tractable computations. Extensive experiments across Gaussian, Poisson, Uniform models, regression with fixed design, and real datasets demonstrate strong robustness to contamination while maintaining competitive performance when well-specified.

Abstract

The -posterior framework provides universal Bayesian estimation with explicit contamination rates and optimal convergence guarantees, but has remained computationally difficult due to an optimization over reference distributions that precludes intractable posterior computation. We develop a PAC-Bayesian framework that recovers these theoretical guarantees through temperature-dependent Gibbs posteriors, deriving finite-sample oracle inequalities with explicit rates and introducing tractable variational approximations that inherit the robustness properties of exact -posteriors. Numerical experiments demonstrate that this approach achieves theoretical contamination rates while remaining computationally feasible, providing the first practical implementation of -posterior inference with rigorous finite-sample guarantees.
Paper Structure (39 sections, 27 theorems, 269 equations, 6 figures, 1 table)

This paper contains 39 sections, 27 theorems, 269 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction and Motivation
  2. Universal Estimation and the Robustness Problem
  3. Robust Bayesian Inference via Generalized Posteriors
  4. The Rho-Posterior Framework: Elegant Theory, Computational Challenge
  5. Sample Hellinger distance.
  6. The contrast function $\psi$.
  7. Pairwise and supremum contrasts.
  8. Main Results
  9. Application to Fixed-Design Regression
  10. Model setup.
  11. Embedding in the independent framework.
  12. Structural assumption.
  13. Regression loss.
  14. Localization and Optimal Rates
  15. Variational Approximation
  16. ...and 24 more sections

Key Result

Lemma 1

For $\psi(x)=(\sqrt{x}-1)/(\sqrt{x}+1)$, there exist universal constants $a_0=4$, $a_1=3/8$, $a_2^2=3\sqrt{2}$ such that for all $(\theta,\theta')\in\Theta^2$, In the i.i.d. case (Remark rem:iid), these bounds hold with $\mathcal{H}_n^2$ replaced by $\mathcal{H}^2$.

Figures (6)

  • Figure 3.1: Gaussian location model results for $n=200$ and $\tau=0.5$. Left: Posterior risk vs. contamination rate $\varepsilon$. Middle: RMSE vs. contamination rate. Right: Posterior densities for a single dataset at $\varepsilon = 10\%$.
  • Figure 3.2: Poisson intensity model results for $n=200$ and $\tau=0.5$. Left: Posterior risk vs. contamination rate $\varepsilon$. Middle: RMSE vs. contamination rate. Right: Posterior densities for a single dataset at $\varepsilon = 10\%$.
  • Figure 3.3: Uniform scale model results for $n=200$ and $\tau=0.5$. Left: Posterior risk vs. contamination rate $\varepsilon$. Middle: RMSE vs. contamination rate. Right: Posterior densities for a single dataset at $\varepsilon = 10\%$.
  • Figure 3.4: Fourier basis regression with smooth target function ($n=200$, $K=6$). Left: Posterior risk versus contamination rate for MLE, Bayes, and $\rho$-posterior. Right: RMSE showing similar robustness hierarchy. The $\rho$-posterior maintains stability while classical methods fail catastrophically.
  • Figure 3.5: Correlated design regression with sparse parameters ($n=100$, $p=11$, $\tau=0.5$). Left: Posterior risk showing OLS catastrophic failure. Middle: RMSE with similar pattern. Right: Predicted versus true values at $\varepsilon=10\%$ demonstrating $\rho$-posterior's ability to recover true regression function.
  • ...and 1 more figures

Theorems & Definitions (55)

  • Remark 1: i.i.d. observations as a special case
  • Lemma 1: Proposition 3 in baraud2018rho
  • Theorem 1: PAC-Bayes bound for independent observations
  • Corollary 1
  • Corollary 2: i.i.d. case
  • proof
  • Corollary 3: Rate with explicit temperature, i.i.d. case
  • Theorem 2: Oracle inequality, i.i.d. case
  • Remark 2: Explicit temperature choice
  • Lemma 2
  • ...and 45 more