Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Block-Sample MAC-Bayes Generalization Bounds

Matthias Frey, Jingge Zhu, Michael C. Gastpar

TL;DR

A family of novel block-sample MAC-Bayes bounds that hold the promise of significantly improving upon the tightness of traditional PAC-Bayes and MAC-Bayes bounds is presented.

Abstract

We present a family of novel block-sample MAC-Bayes bounds (mean approximately correct). While PAC-Bayes bounds (probably approximately correct) typically give bounds for the generalization error that hold with high probability, MAC-Bayes bounds have a similar form but bound the expected generalization error instead. The family of bounds we propose can be understood as a generalization of an expectation version of known PAC-Bayes bounds. Compared to standard PAC-Bayes bounds, the new bounds contain divergence terms that only depend on subsets (or \emph{blocks}) of the training data. The proposed MAC-Bayes bounds hold the promise of significantly improving upon the tightness of traditional PAC-Bayes and MAC-Bayes bounds. This is illustrated with a simple numerical example in which the original PAC-Bayes bound is vacuous regardless of the choice of prior, while the proposed family of bounds are finite for appropriate choices of the block size. We also explore the question whether high-probability versions of our MAC-Bayes bounds (i.e., PAC-Bayes bounds of a similar form) are possible. We answer this question in the negative with an example that shows that in general, it is not possible to establish a PAC-Bayes bound which (a) vanishes with a rate faster than $\mathcal{O}(1/\log n)$ whenever the proposed MAC-Bayes bound vanishes with rate $\mathcal{O}(n^{-1/2})$ and (b) exhibits a logarithmic dependence on the permitted error probability.

Block-Sample MAC-Bayes Generalization Bounds

TL;DR

A family of novel block-sample MAC-Bayes bounds that hold the promise of significantly improving upon the tightness of traditional PAC-Bayes and MAC-Bayes bounds is presented.

Abstract

We present a family of novel block-sample MAC-Bayes bounds (mean approximately correct). While PAC-Bayes bounds (probably approximately correct) typically give bounds for the generalization error that hold with high probability, MAC-Bayes bounds have a similar form but bound the expected generalization error instead. The family of bounds we propose can be understood as a generalization of an expectation version of known PAC-Bayes bounds. Compared to standard PAC-Bayes bounds, the new bounds contain divergence terms that only depend on subsets (or \emph{blocks}) of the training data. The proposed MAC-Bayes bounds hold the promise of significantly improving upon the tightness of traditional PAC-Bayes and MAC-Bayes bounds. This is illustrated with a simple numerical example in which the original PAC-Bayes bound is vacuous regardless of the choice of prior, while the proposed family of bounds are finite for appropriate choices of the block size. We also explore the question whether high-probability versions of our MAC-Bayes bounds (i.e., PAC-Bayes bounds of a similar form) are possible. We answer this question in the negative with an example that shows that in general, it is not possible to establish a PAC-Bayes bound which (a) vanishes with a rate faster than whenever the proposed MAC-Bayes bound vanishes with rate and (b) exhibits a logarithmic dependence on the permitted error probability.
Paper Structure (19 sections, 6 theorems, 63 equations, 1 figure, 1 table)

This paper contains 19 sections, 6 theorems, 63 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related works
  3. Block-sample MAC-Bayes bounds
  4. Example: Gaussian mean estimation with truncated loss function
  5. Optimizing block-sample MAC-Bayes bounds
  6. On the (im)possibility of block-sample PAC-Bayes bounds
  7. Conclusions and limitations
  8. Proof details for Corollaries and Remarks in Section \ref{['sec:main_result']}
  9. Proof of Corollary \ref{['cor:catoni']}
  10. Calculation Details for Remark \ref{['remark:kl']}
  11. Proof of Corollary \ref{['cor:difference']}
  12. Detailed argument for Remark \ref{['remark:deterministic-algorithm']}
  13. Calculations for Table \ref{['table:comparison_general']} in Section \ref{['section:optimizing']}
  14. Proof of Theorem \ref{['thm:counterexample']}
  15. Choice of $\ell$, $P_Z$, and $P_{W|S}$.
  16. ...and 4 more sections

Key Result

Theorem 1

Let $m \in [n] := \{1, \dots, n\}$, assume that $n$ is an integer multiple of $m$, and let $S'=(Z'_1,\ldots, Z'_{m})$ where $Z'_1\ldots, Z'_m$ are i.i.d. drawn from $P_Z$. Assume that for $\lambda'\in (0,b)$ and some distribution $Q_W$ over $\mathcal{W}$, it holds that for some function $\Phi_m: (0,b) \rightarrow (0,\infty)$. Then for any $\lambda \in (0,bn/m)$, it holds that where $J := n/m$.

Figures (1)

  • Figure 1: Comparison of true generalization error and theoretical bounds for the example in Section \ref{['sec:gaussian-truncated-loss']} with $\mu=1/2$. The solid blue curve shows the optimal bound \ref{['eq:example-catoni-bound']} with optimal choice for $m$.

Theorems & Definitions (14)

  • Theorem 1: Block-sample MAC-Bayes bounds
  • proof
  • Remark 1
  • Remark 2: Data dependent priors
  • Corollary 1
  • Remark 3
  • Corollary 2
  • Remark 4
  • Theorem 2
  • Remark 5
  • ...and 4 more