Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Symmetry and Generalisation in Machine Learning

Hayder Elesedy

TL;DR

This thesis develops a unified, operator-based view of symmetry in machine learning, showing that averaging over symmetry groups yields an equivariant or invariant predictor with lower or equal risk under squared-loss tasks when the symmetry is correct. Central to the theory is the averaging operator ${\mathcal Q}$ (and ${\mathcal O}$ for invariance), which decomposes any predictor into a symmetry-respecting component and an orthogonal anti-symmetric part, with the generalisation gap equalling the anti-symmetric norm. The results are instantiated in linear models and kernel methods, providing explicit expressions for generalisation gains in terms of group structure, data, and model properties, and extended to orbit-representative viewpoints that reduce learning to a cross-section problem. The work also connects these viewpoints to neural networks, offering guidance on architectural design (via weight-tying and regularisation) and highlighting opportunities for data-driven symmetry discovery. Overall, the findings offer a rigorous, practically relevant framework for leveraging symmetry to improve generalisation in diverse learning settings, including random-design least squares and kernel ridge regression, while revealing deep links between orbit averaging and orbit-representative approaches.

Abstract

This work is about understanding the impact of invariance and equivariance on generalisation in supervised learning. We use the perspective afforded by an averaging operator to show that for any predictor that is not equivariant, there is an equivariant predictor with strictly lower test risk on all regression problems where the equivariance is correctly specified. This constitutes a rigorous proof that symmetry, in the form of invariance or equivariance, is a useful inductive bias. We apply these ideas to equivariance and invariance in random design least squares and kernel ridge regression respectively. This allows us to specify the reduction in expected test risk in more concrete settings and express it in terms of properties of the group, the model and the data. Along the way, we give examples and additional results to demonstrate the utility of the averaging operator approach in analysing equivariant predictors. In addition, we adopt an alternative perspective and formalise the common intuition that learning with invariant models reduces to a problem in terms of orbit representatives. The formalism extends naturally to a similar intuition for equivariant models. We conclude by connecting the two perspectives and giving some ideas for future work.

Symmetry and Generalisation in Machine Learning

TL;DR

This thesis develops a unified, operator-based view of symmetry in machine learning, showing that averaging over symmetry groups yields an equivariant or invariant predictor with lower or equal risk under squared-loss tasks when the symmetry is correct. Central to the theory is the averaging operator (and for invariance), which decomposes any predictor into a symmetry-respecting component and an orthogonal anti-symmetric part, with the generalisation gap equalling the anti-symmetric norm. The results are instantiated in linear models and kernel methods, providing explicit expressions for generalisation gains in terms of group structure, data, and model properties, and extended to orbit-representative viewpoints that reduce learning to a cross-section problem. The work also connects these viewpoints to neural networks, offering guidance on architectural design (via weight-tying and regularisation) and highlighting opportunities for data-driven symmetry discovery. Overall, the findings offer a rigorous, practically relevant framework for leveraging symmetry to improve generalisation in diverse learning settings, including random-design least squares and kernel ridge regression, while revealing deep links between orbit averaging and orbit-representative approaches.

Abstract

This work is about understanding the impact of invariance and equivariance on generalisation in supervised learning. We use the perspective afforded by an averaging operator to show that for any predictor that is not equivariant, there is an equivariant predictor with strictly lower test risk on all regression problems where the equivariance is correctly specified. This constitutes a rigorous proof that symmetry, in the form of invariance or equivariance, is a useful inductive bias. We apply these ideas to equivariance and invariance in random design least squares and kernel ridge regression respectively. This allows us to specify the reduction in expected test risk in more concrete settings and express it in terms of properties of the group, the model and the data. Along the way, we give examples and additional results to demonstrate the utility of the averaging operator approach in analysing equivariant predictors. In addition, we adopt an alternative perspective and formalise the common intuition that learning with invariant models reduces to a problem in terms of orbit representatives. The formalism extends naturally to a similar intuition for equivariant models. We conclude by connecting the two perspectives and giving some ideas for future work.
Paper Structure (96 sections, 60 theorems, 257 equations, 1 figure)

This paper contains 96 sections, 60 theorems, 257 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Motivation
  3. Overview
  4. Authorship
  5. Preliminaries
  6. Setup, Assumptions and Technicalities
  7. Spaces
  8. Group and Action
  9. Invariance, Equivariance and Symmetry
  10. Function Space
  11. Averaging Operator
  12. Additional Notation
  13. Commonly Used Results
  14. The inclusion ${L_2(\mu)} \subset L_1(\mu)$
  15. Fubini's theorem
  16. ...and 81 more sections

Key Result

Theorem 1.1

Let $(\mathcal{X}, \mathcal{S}_\mathcal{X}, \mu)$ be a measure space with $\mu \ge 0$. Let $A_\infty = \{A \in \mathcal{S}_\mathcal{X}: \mu(A) < \infty\}$. Then $\sup_{A\in A_\infty} \mu(A)< \infty$ is equivalent to $L_p(\mu) \subset L_q(\mu)$ for all $p > q$.

Figures (1)

  • Figure 1: Example of a function decomposition. The figure shows $f(r, \theta) = r \cos{(r - 2\theta)} \cos{(r + 2\theta)}$ decomposed into its $\mathcal{G}$-symmetric and $\mathcal{G}$-anti-symmetric parts in $V = S \oplus A$ under the natural action of $\mathcal{G} = \mathsf{SO}_2$ on $\mathbb{R}^2$. See \ref{['example:rotations']}. Image credit: Sheheryar Zaidi. Best viewed in colour.

Theorems & Definitions (120)

  • Theorem 1.1: villiani1985LpLq
  • Theorem 1.2: Fubini's theorem kallenberg2006foundations
  • Theorem 1.3: folland2016course
  • Theorem 1.4: Haar measure kallenberg2006foundations
  • Theorem 2.1: lehmann2005testing
  • proof
  • Lemma 3.1
  • Example 3.2
  • Remark 3.3
  • Lemma : \ref{['lemma:l2-decomposition']}
  • ...and 110 more