Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Symmetries in PAC-Bayesian Learning

Armin Beck, Peter Ochs

TL;DR

The paper extends PAC-Bayesian generalization guarantees to non-compact symmetries and non-invariant data distributions by introducing a group-averaging operator that projects hypotheses onto the equivariant subset. It proves a KL-divergence decomposition, enabling tighter McAllester-type PAC-Bayes bounds via the pushforward measures, and shows that risk and empirical risk can be characterized on a reduced set of orbit representatives when data and loss respect the symmetry. The approach is validated on rotated MNIST with non-uniform rotations (and translations), where an equivariant CNN achieves better empirical performance and a strictly tighter bound than a baseline non-equivariant model. This work broadens the theoretical foundations of symmetry in learning and provides practical guidance for leveraging symmetry beyond compact groups and invariant data distributions.

Abstract

Symmetries are known to improve the empirical performance of machine learning models, yet theoretical guarantees explaining these gains remain limited. Prior work has focused mainly on compact group symmetries and often assumes that the data distribution itself is invariant, an assumption rarely satisfied in real-world applications. In this work, we extend generalization guarantees to the broader setting of non-compact symmetries, such as translations and to non-invariant data distributions. Building on the PAC-Bayes framework, we adapt and tighten existing bounds, demonstrating the approach on McAllester's PAC-Bayes bound while showing that it applies to a wide range of PAC-Bayes bounds. We validate our theory with experiments on a rotated MNIST dataset with a non-uniform rotation group, where the derived guarantees not only hold but also improve upon prior results. These findings provide theoretical evidence that, for symmetric data, symmetric models are preferable beyond the narrow setting of compact groups and invariant distributions, opening the way to a more general understanding of symmetries in machine learning.

Symmetries in PAC-Bayesian Learning

TL;DR

The paper extends PAC-Bayesian generalization guarantees to non-compact symmetries and non-invariant data distributions by introducing a group-averaging operator that projects hypotheses onto the equivariant subset. It proves a KL-divergence decomposition, enabling tighter McAllester-type PAC-Bayes bounds via the pushforward measures, and shows that risk and empirical risk can be characterized on a reduced set of orbit representatives when data and loss respect the symmetry. The approach is validated on rotated MNIST with non-uniform rotations (and translations), where an equivariant CNN achieves better empirical performance and a strictly tighter bound than a baseline non-equivariant model. This work broadens the theoretical foundations of symmetry in learning and provides practical guidance for leveraging symmetry beyond compact groups and invariant data distributions.

Abstract

Symmetries are known to improve the empirical performance of machine learning models, yet theoretical guarantees explaining these gains remain limited. Prior work has focused mainly on compact group symmetries and often assumes that the data distribution itself is invariant, an assumption rarely satisfied in real-world applications. In this work, we extend generalization guarantees to the broader setting of non-compact symmetries, such as translations and to non-invariant data distributions. Building on the PAC-Bayes framework, we adapt and tighten existing bounds, demonstrating the approach on McAllester's PAC-Bayes bound while showing that it applies to a wide range of PAC-Bayes bounds. We validate our theory with experiments on a rotated MNIST dataset with a non-uniform rotation group, where the derived guarantees not only hold but also improve upon prior results. These findings provide theoretical evidence that, for symmetric data, symmetric models are preferable beyond the narrow setting of compact groups and invariant distributions, opening the way to a more general understanding of symmetries in machine learning.
Paper Structure (18 sections, 13 theorems, 43 equations, 1 figure)

This paper contains 18 sections, 13 theorems, 43 equations, 1 figure.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Preliminaries and Notations
  4. SYMMETRY IN HYPOTHESIS CLASS
  5. PAC-Bayesian Bound
  6. SYMMETRY ON DATA
  7. Evaluating Risk on Representatives
  8. NUMERICAL ANALYSIS
  9. Data Generation.
  10. Models.
  11. Training.
  12. Results.
  13. CONCLUSION
  14. APPENDIX: COLLECTION OF PROOFS
  15. Proof of Proposition \ref{['proposition averaged risk']}
  16. ...and 3 more sections

Key Result

Theorem 1.1

kallenberg2002foundations Let $(\mathcal{S}, \mathscr{S})$ and $(\mathcal{T}, \mathscr{T})$ be measurable spaces, where $\mathcal{T}$ is Borel. Let $\mu$ be a probability measure on $\mathcal{S} \times \mathcal{T}$. Then $\mu = \mu_\mathcal{S} \otimes \kappa$, where $\mu_\mathcal{S} \equiv \mu(\cdot

Figures (1)

  • Figure 1: The figure displays the distribution of the test errors for the baseline CNN and the equivariant CNN on two datasets: the rotated MNIST (left) and the rotated and translated MNIST (right). Our PAC-Bayesian bound (Theorem \ref{['theorem final PAC-bound']}) is visualized with a vertical dash-dotted line labeled as McAllester bound. The figure highlights two key observations. First, the equivariant CNN outperforms the baseline CNN in terms of empirical accuracy on rotated MNIST. Second, and more importantly from a theoretical perspective, the equivariant network also enjoys a strictly smaller generalization bound that is closer to the test error.

Theorems & Definitions (29)

  • Theorem 1.1
  • Theorem 1.2
  • Lemma 1.3
  • Definition 2.2
  • Remark 2.3
  • Theorem 2.5
  • proof
  • Theorem 2.6
  • Remark 2.7
  • Remark 3.2
  • ...and 19 more