Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Generalization Error of Supervised Machine Learning Algorithms

Samir M. Perlaza, Xinying Zou

TL;DR

The paper develops the method of gaps to obtain exact, closed-form expressions for the generalization error in supervised learning using information-theoretic quantities. It reveals two complementary pathways: algorithm-driven gaps based on Gibbs algorithms and data-driven gaps based on worst-case data-generating measures, each yielding tractable expressions in terms of relative entropies, mutual information, and lautum information. A key outcome is that the Gibbs path gives a clean formula for the Gibbs algorithm: $\overline{\overline{\mathsf{G}}} = \lambda\big(I(\cdot;\cdot) + L(\cdot;\cdot)\big)$, while the data-driven path connects to mismatched hypothesis testing and a spectrum of geometric interpretations. The results unify known exact generalization expressions and generate new ones, providing deep insights into how algorithm-data interactions shape generalization; these foundations illuminate connections to hypothesis testing and geometry and pave the way for extensions to federated and other complex learning paradigms.

Abstract

In this paper, the method of gaps, a technique for deriving closed-form expressions in terms of information measures for the generalization error of supervised machine learning algorithms is introduced. The method relies on the notion of \emph{gaps}, which characterize the variation of the expected empirical risk (when either the model or dataset is kept fixed) with respect to changes in the probability measure on the varying parameter (either the dataset or the model, respectively). This distinction results in two classes of gaps: Algorithm-driven gaps (fixed dataset) and data-driven gaps (fixed model). In general, the method relies on two central observations: $(i)$~The generalization error is the expectation of an algorithm-driven gap or a data-driven gap. In the first case, the expectation is with respect to a measure on the datasets; and in the second case, with respect to a measure on the models. $(ii)$~Both, algorithm-driven gaps and data-driven gaps exhibit closed-form expressions in terms of relative entropies. In particular, algorithm-driven gaps involve a Gibbs probability measure on the set of models, which represents a supervised Gibbs algorithm. Alternatively, data-driven gaps involve a worst-case data-generating (WCDG) probability measure on the set of data points, which is also a Gibbs probability measure. Interestingly, such Gibbs measures, which are exogenous to the analysis of generalization, place both the supervised Gibbs algorithm and the WCDG probability measure as natural references for the analysis of supervised learning algorithms. All existing exact expressions for the generalization error of supervised machine learning algorithms can be obtained with the proposed method. Also, this method allows obtaining numerous new exact expressions, which allows establishing connections with other areas in statistics.

The Generalization Error of Supervised Machine Learning Algorithms

TL;DR

The paper develops the method of gaps to obtain exact, closed-form expressions for the generalization error in supervised learning using information-theoretic quantities. It reveals two complementary pathways: algorithm-driven gaps based on Gibbs algorithms and data-driven gaps based on worst-case data-generating measures, each yielding tractable expressions in terms of relative entropies, mutual information, and lautum information. A key outcome is that the Gibbs path gives a clean formula for the Gibbs algorithm: , while the data-driven path connects to mismatched hypothesis testing and a spectrum of geometric interpretations. The results unify known exact generalization expressions and generate new ones, providing deep insights into how algorithm-data interactions shape generalization; these foundations illuminate connections to hypothesis testing and geometry and pave the way for extensions to federated and other complex learning paradigms.

Abstract

In this paper, the method of gaps, a technique for deriving closed-form expressions in terms of information measures for the generalization error of supervised machine learning algorithms is introduced. The method relies on the notion of \emph{gaps}, which characterize the variation of the expected empirical risk (when either the model or dataset is kept fixed) with respect to changes in the probability measure on the varying parameter (either the dataset or the model, respectively). This distinction results in two classes of gaps: Algorithm-driven gaps (fixed dataset) and data-driven gaps (fixed model). In general, the method relies on two central observations: ~The generalization error is the expectation of an algorithm-driven gap or a data-driven gap. In the first case, the expectation is with respect to a measure on the datasets; and in the second case, with respect to a measure on the models. ~Both, algorithm-driven gaps and data-driven gaps exhibit closed-form expressions in terms of relative entropies. In particular, algorithm-driven gaps involve a Gibbs probability measure on the set of models, which represents a supervised Gibbs algorithm. Alternatively, data-driven gaps involve a worst-case data-generating (WCDG) probability measure on the set of data points, which is also a Gibbs probability measure. Interestingly, such Gibbs measures, which are exogenous to the analysis of generalization, place both the supervised Gibbs algorithm and the WCDG probability measure as natural references for the analysis of supervised learning algorithms. All existing exact expressions for the generalization error of supervised machine learning algorithms can be obtained with the proposed method. Also, this method allows obtaining numerous new exact expressions, which allows establishing connections with other areas in statistics.

Paper Structure

This paper contains 42 sections, 39 theorems, 10 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Gaps and the Method of Gaps
  3. Notation
  4. Statistical Supervised Machine Learning
  5. Generalization Error
  6. The Method of Gaps
  7. Gaps
  8. Method of Algorithm-driven Gaps
  9. Step One
  10. Step Two
  11. Method of Data-driven Gaps
  12. Step One
  13. Step Two
  14. Explicit Expressions Obtained Via the Method of Algorithm-Driven Gaps
  15. Connections to Statistical Hypothesis Testing
  16. ...and 27 more sections

Key Result

Lemma 1

The generalization error $\overline{\overline{\mathsf{G}}} \left(P_{\boldsymbol{\Theta} | \boldsymbol{Z}}, P_{\boldsymbol{Z}} \right)$ in EqJun4at9h02in2024 satisfies where the functional $\mathsf{R}_{\boldsymbol{\theta}}$ is defined in EqRModel and the function $\mathsf{L}$ is defined in EqLxy.

Figures (7)

  • Figure 1: Geometric interpretation of Lemma \ref{['LemmaSeptember4at17h56in2024']} involving only relative entropies. Note that $\int D\left( P_{\boldsymbol{\Theta}| \boldsymbol{Z} = \boldsymbol{z}} \| P_{\boldsymbol{\Theta}}\right) \mathrm{d}P_{\boldsymbol{Z}} \left( \boldsymbol{z} \right) = I\left(P_{\boldsymbol{\Theta}| \boldsymbol{Z}}; P_{\boldsymbol{Z}} \right)$.
  • Figure 2: Geometric interpretation of Lemma \ref{['LemmaSeptember4at17h56in2024']} involving the generalization error $\overline{\overline{\mathsf{G}}} \left(P_{\boldsymbol{\Theta} | \boldsymbol{Z}}, P_{\boldsymbol{Z}} \right)$ in \ref{['EqJun4at9h02in2024']}.
  • Figure 3: Geometric interpretation of Lemma \ref{['LemmaSeptember4at17h56in2024']} for the Gibbs algorithm $P^{\left(Q, \lambda\right)}_{\boldsymbol{\Theta}| \boldsymbol{Z}}$ (Definition \ref{['DefGibbsAlgorithm']}).
  • Figure 4: Geometric interpretation of Lemma \ref{['LemmaGeometricEntropy']} involving only relative entropies. Note that $\int D\left( P_{Z|\boldsymbol{\Theta}=\boldsymbol{\theta}} \| P_Z\right)\mathrm{d}P_{\boldsymbol{\Theta}}(\boldsymbol{\theta}) = I\left(P_{Z|\boldsymbol{\Theta}}; P_{\boldsymbol{\Theta}} \right)$.
  • Figure 5: Geometric interpretation of Lemma \ref{['LemmaGeometricEntropy']} involving only relative entropies. Note that $\int D\left( P_{Z|\boldsymbol{\Theta}=\boldsymbol{\theta}} \| P_Z\right)\mathrm{d}P_{\boldsymbol{\Theta}}(\boldsymbol{\theta}) = I\left(P_{Z|\boldsymbol{\Theta}}; P_{\boldsymbol{\Theta}} \right)$.
  • ...and 2 more figures

Theorems & Definitions (45)

  • Definition 1: Relative Entropy
  • Definition 2: Algorithm
  • Definition 3: Empirical Risk
  • Definition 4: Generalization Error
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Definition 5: Gibbs Algorithm
  • Lemma 4
  • Lemma 5
  • ...and 35 more