Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Rates of Empirical Risk Minimization

Steve Hanneke, Mingyue Xu

TL;DR

This work analyzes universal learning by ERM in the realizable setting and proves a four-way dichotomy: the universal ERM learning rate must be $e^{-n}$, $1/n$, $\log(n)/n$, or arbitrarily slow, depending on distribution-dependent complexity structures. It develops three eluder-based sequences—eluder, star-eluder, and VC-eluder—and two associated dimensions (SE and VCE) to exactly characterize when each regime occurs and to provide sharp constants when possible. The results connect to classical complexity measures (VC and Littlestone dimensions) while revealing cases where ERM is not optimal among universal learners, and they include target-dependent refinements distinguishing the rate achievable for specific target concepts. These insights lay groundwork for extensions to agnostic, multiclass, and interactive settings and illuminate the nuanced performance of ERM beyond worst-case PAC theory.

Abstract

The well-known empirical risk minimization (ERM) principle is the basis of many widely used machine learning algorithms, and plays an essential role in the classical PAC theory. A common description of a learning algorithm's performance is its so-called "learning curve", that is, the decay of the expected error as a function of the input sample size. As the PAC model fails to explain the behavior of learning curves, recent research has explored an alternative universal learning model and has ultimately revealed a distinction between optimal universal and uniform learning rates (Bousquet et al., 2021). However, a basic understanding of such differences with a particular focus on the ERM principle has yet to be developed. In this paper, we consider the problem of universal learning by ERM in the realizable case and study the possible universal rates. Our main result is a fundamental tetrachotomy: there are only four possible universal learning rates by ERM, namely, the learning curves of any concept class learnable by ERM decay either at $e^{-n}$, $1/n$, $\log(n)/n$, or arbitrarily slow rates. Moreover, we provide a complete characterization of which concept classes fall into each of these categories, via new complexity structures. We also develop new combinatorial dimensions which supply sharp asymptotically-valid constant factors for these rates, whenever possible.

Universal Rates of Empirical Risk Minimization

TL;DR

This work analyzes universal learning by ERM in the realizable setting and proves a four-way dichotomy: the universal ERM learning rate must be , , , or arbitrarily slow, depending on distribution-dependent complexity structures. It develops three eluder-based sequences—eluder, star-eluder, and VC-eluder—and two associated dimensions (SE and VCE) to exactly characterize when each regime occurs and to provide sharp constants when possible. The results connect to classical complexity measures (VC and Littlestone dimensions) while revealing cases where ERM is not optimal among universal learners, and they include target-dependent refinements distinguishing the rate achievable for specific target concepts. These insights lay groundwork for extensions to agnostic, multiclass, and interactive settings and illuminate the nuanced performance of ERM beyond worst-case PAC theory.

Abstract

The well-known empirical risk minimization (ERM) principle is the basis of many widely used machine learning algorithms, and plays an essential role in the classical PAC theory. A common description of a learning algorithm's performance is its so-called "learning curve", that is, the decay of the expected error as a function of the input sample size. As the PAC model fails to explain the behavior of learning curves, recent research has explored an alternative universal learning model and has ultimately revealed a distinction between optimal universal and uniform learning rates (Bousquet et al., 2021). However, a basic understanding of such differences with a particular focus on the ERM principle has yet to be developed. In this paper, we consider the problem of universal learning by ERM in the realizable case and study the possible universal rates. Our main result is a fundamental tetrachotomy: there are only four possible universal learning rates by ERM, namely, the learning curves of any concept class learnable by ERM decay either at , , , or arbitrarily slow rates. Moreover, we provide a complete characterization of which concept classes fall into each of these categories, via new complexity structures. We also develop new combinatorial dimensions which supply sharp asymptotically-valid constant factors for these rates, whenever possible.

Paper Structure

This paper contains 24 sections, 44 theorems, 91 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Basic examples
  3. Main results
  4. Related works
  5. Technical overview
  6. Exact universal rates
  7. Exponential rates
  8. Linear rates
  9. $\log{(n)}/n$ rates
  10. Arbitrarily slow rates
  11. Equivalent characterizations
  12. Appendix Summary
  13. Conclusion and Future Directions
  14. Preliminaries
  15. Detailed examples
  16. ...and 9 more sections

Key Result

Theorem 1

For every class $\mathcal{H}$ with $|\mathcal{H}|\geq 3$, the following hold:

Figures (1)

  • Figure 1: A venn diagram depicting the tetrachotomy of the universal rates by ERM and its relation with the uniform rates characterized by the VC dimension and the star number.

Theorems & Definitions (128)

  • Definition 1: mitchell1977version
  • Definition 2: Universal learning by ERM
  • Remark 1
  • Remark 2
  • Example 1: $e^{-n}$ learning rate
  • Example 2: $1/n$ learning rate
  • Example 3: $\log{(n)}/n$ learning rate
  • Example 4: Arbitrarily slow learning rate
  • Example 5: Not Glivenko-Cantelli but learnable by ERM
  • Theorem 1: Universal rates for ERM
  • ...and 118 more