Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization Certificates for Adversarially Robust Bayesian Linear Regression

Mahalakshmi Sabanayagam, Russell Tsuchida, Cheng Soon Ong, Debarghya Ghoshdastidar

TL;DR

The paper tackles adversarial perturbations in probabilistic inference by defining adversarially robust posteriors via an adversarially perturbed negative log-likelihood within a generalized Bayesian framework. It shows how, for exponential-family models, the adversarial NLL admits closed-form expressions, enabling a tractable robust posterior $q_\delta(\theta)$. Leveraging the PAC-Bayesian framework, the authors derive the first rigorous generalization certificates for both standard and adversarial posteriors in Bayesian linear regression, with extensions to robust settings and explicit bound terms. Experiments on real and synthetic data corroborate the theoretical claims, demonstrating enhanced adversarial robustness of $q_\delta$ over the standard Bayes posterior and validating the proposed certificates as meaningful predictors of generalization performance in adversarial regimes.

Abstract

Adversarial robustness of machine learning models is critical to ensuring reliable performance under data perturbations. Recent progress has been on point estimators, and this paper considers distributional predictors. First, using the link between exponential families and Bregman divergences, we formulate an adversarial Bregman divergence loss as an adversarial negative log-likelihood. Using the geometric properties of Bregman divergences, we compute the adversarial perturbation for such models in closed-form. Second, under such losses, we introduce \emph{adversarially robust posteriors}, by exploiting the optimization-centric view of generalized Bayesian inference. Third, we derive the \emph{first} rigorous generalization certificates in the context of an adversarial extension of Bayesian linear regression by leveraging the PAC-Bayesian framework. Finally, experiments on real and synthetic datasets demonstrate the superior robustness of the derived adversarially robust posterior over Bayes posterior, and also validate our theoretical guarantees.

Generalization Certificates for Adversarially Robust Bayesian Linear Regression

TL;DR

The paper tackles adversarial perturbations in probabilistic inference by defining adversarially robust posteriors via an adversarially perturbed negative log-likelihood within a generalized Bayesian framework. It shows how, for exponential-family models, the adversarial NLL admits closed-form expressions, enabling a tractable robust posterior . Leveraging the PAC-Bayesian framework, the authors derive the first rigorous generalization certificates for both standard and adversarial posteriors in Bayesian linear regression, with extensions to robust settings and explicit bound terms. Experiments on real and synthetic data corroborate the theoretical claims, demonstrating enhanced adversarial robustness of over the standard Bayes posterior and validating the proposed certificates as meaningful predictors of generalization performance in adversarial regimes.

Abstract

Adversarial robustness of machine learning models is critical to ensuring reliable performance under data perturbations. Recent progress has been on point estimators, and this paper considers distributional predictors. First, using the link between exponential families and Bregman divergences, we formulate an adversarial Bregman divergence loss as an adversarial negative log-likelihood. Using the geometric properties of Bregman divergences, we compute the adversarial perturbation for such models in closed-form. Second, under such losses, we introduce \emph{adversarially robust posteriors}, by exploiting the optimization-centric view of generalized Bayesian inference. Third, we derive the \emph{first} rigorous generalization certificates in the context of an adversarial extension of Bayesian linear regression by leveraging the PAC-Bayesian framework. Finally, experiments on real and synthetic datasets demonstrate the superior robustness of the derived adversarially robust posterior over Bayes posterior, and also validate our theoretical guarantees.

Paper Structure

This paper contains 47 sections, 15 theorems, 50 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation
  4. Adversarial robustness
  5. Probabilistic inference
  6. Bayesian inference
  7. Generalized Bayesian inference
  8. Gibbs Bayesian inference
  9. Gaussian linear regression
  10. Linking loss functions and probability distributions
  11. Exponential families
  12. Bregman divergence
  13. Adversarially robust generalized linear models
  14. Adversarial negative log likelihood
  15. Standard and Adversarial Generalization Certificates
  16. ...and 32 more sections

Key Result

Lemma 3.1

Under a linear predictor $f_\theta(x)=\theta^\top x$ and in the case where the exponential family is a Gaussian family, and the adversarial perturbation of the sample $x$ is $\widetilde{x} = \delta \mathop{\mathrm{sign}}\nolimits(\theta^\top x - y) \frac{\theta}{\Vert \theta \Vert_2} + x = \delta \mathop{\mathrm{sign}}\nolimits(\theta^\top \widetilde{x} - y) \frac{\theta}{\Vert \theta \Vert_2} +

Figures (1)

  • Figure 1: Validation of the derived generalization certificates \ref{['thm:std_post_std_loss', 'thm:std_post_adv_loss', 'thm:adv_post_std_loss', 'thm:adv_post_adv_loss']}. (left to right) Standard generalization of Bayes posterior with comparison to prior work germain2016pac, standard generalization of the robust posterior, adversarial generalization of Bayes posterior, and adversarial generalization of the robust posterior.

Theorems & Definitions (19)

  • Lemma 3.1: Robust loss in closed-form for Gaussian likelihood
  • Lemma 3.2: Robust loss in closed-form for exponential family likelihood
  • Corollary 3.3: Robust posterior
  • Theorem 4.1: Theorem 6 in banerjee2021information
  • Lemma 4.2: CGF bounds for standard and adversarial losses
  • Theorem 4.3: Standard generalization of Bayes posterior, adapted from germain2016pac
  • Theorem 4.4: Adversarial generalization of Bayes posterior
  • Theorem 4.5: Standard generalization of robust posterior
  • Theorem 4.6: Adversarial generalization of robust posterior with $\hat{\delta}=\delta$
  • Theorem 4.7: Adversarial generalization of robust posterior
  • ...and 9 more