Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A note on generalization bounds for losses with finite moments

Borja Rodríguez-Gálvez, Omar Rivasplata, Ragnar Thobaben, Mikael Skoglund

TL;DR

A high-probability PAC-Bayes bound for losses with a bounded variance is derived, which has an exponentially better dependence on the confidence parameter and the dependency measure than previous bounds in the literature.

Abstract

This paper studies the truncation method from Alquier [1] to derive high-probability PAC-Bayes bounds for unbounded losses with heavy tails. Assuming that the $p$-th moment is bounded, the resulting bounds interpolate between a slow rate $1 / \sqrt{n}$ when $p=2$, and a fast rate $1 / n$ when $p \to \infty$ and the loss is essentially bounded. Moreover, the paper derives a high-probability PAC-Bayes bound for losses with a bounded variance. This bound has an exponentially better dependence on the confidence parameter and the dependency measure than previous bounds in the literature. Finally, the paper extends all results to guarantees in expectation and single-draw PAC-Bayes. In order to so, it obtains analogues of the PAC-Bayes fast rate bound for bounded losses from [2] in these settings.

A note on generalization bounds for losses with finite moments

TL;DR

A high-probability PAC-Bayes bound for losses with a bounded variance is derived, which has an exponentially better dependence on the confidence parameter and the dependency measure than previous bounds in the literature.

Abstract

This paper studies the truncation method from Alquier [1] to derive high-probability PAC-Bayes bounds for unbounded losses with heavy tails. Assuming that the -th moment is bounded, the resulting bounds interpolate between a slow rate when , and a fast rate when and the loss is essentially bounded. Moreover, the paper derives a high-probability PAC-Bayes bound for losses with a bounded variance. This bound has an exponentially better dependence on the confidence parameter and the dependency measure than previous bounds in the literature. Finally, the paper extends all results to guarantees in expectation and single-draw PAC-Bayes. In order to so, it obtains analogues of the PAC-Bayes fast rate bound for bounded losses from [2] in these settings.
Paper Structure (17 sections, 17 theorems, 56 equations, 1 figure)

This paper contains 17 sections, 17 theorems, 56 equations, 1 figure.

Table of Contents

  1. Introduction
  2. PAC-Bayesian bounds
  3. Contributions
  4. Alquier's truncation method
  5. Refining the method
  6. Losses with a bounded moment
  7. Alquier's modification for losses with a bounded moment
  8. Optimizing the parameter in the bound
  9. The case and essentially bounded losses
  10. Losses with a bounded variance
  11. Extension of the results
  12. Omitted proofs
  13. Proof of Theorem 2
  14. Proof of Theorem 5
  15. Extension to bounds in expectation
  16. ...and 2 more sections

Key Result

Lemma 1

For all $\beta \in (0,1)$ and all $\lambda > 0$, with probability no smaller than $1 - \beta$, holds simultaneously$\forall (\mathbb{P}_W^S, c, \gamma) \in \mathcal{P} \times (0, 1] \times [1, \infty)$.

Figures (1)

  • Figure 1: Illustration comparing alquier2018simplerohnishi2021novel (\ref{['eq:bounded_variance_alquier']} in black, \ref{['eq:bounded_variance_honorio_1']} in gray, and \ref{['eq:bounded_variance_honorio_2']} in orange) and our \ref{['th:bounded_variance_high_probability']} (in blue) for varying values of $\beta$, $\chi^2$, $\widehat{\mathscr{R}}$, and $n$. To help the comparison, we actually use the upper bound relaxation \ref{['eq:relaxation_of_bounded_variance_high_probability']} of \ref{['th:bounded_variance_high_probability']}. When they are not varying, the values of the parameters are fixed to $\beta = 0.025$, $\chi^2 = 200$, $\widehat{\mathscr{R}} = 0.025$, $n=10,000$, and $\sigma^2 = 1$.

Theorems & Definitions (21)

  • Lemma 1: alquier2006transductive
  • Lemma 2: Refinement of \ref{['lemma:alquier_truncation_method']}
  • Lemma 3
  • Lemma 4: alquier2006transductive
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4: alquier2018simpler and ohnishi2021novel
  • Theorem 5
  • proof : Sketch of the proof
  • ...and 11 more