Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Complexity of Agnostic Multiclass Classification: Natarajan Dimension Strikes Back

Alon Cohen, Liad Erez, Steve Hanneke, Tomer Koren, Yishay Mansour, Shay Moran, Qian Zhang

TL;DR

It is shown that agnostic multiclass PAC sample complexity is in fact governed by two distinct dimensions, and it is proved nearly tight agnostic sample complexity bounds that, up to log factors, take the form of $\epsilon$ where $\epsilon$ is the excess risk.

Abstract

The fundamental theorem of statistical learning states that binary PAC learning is governed by a single parameter -- the Vapnik-Chervonenkis (VC) dimension -- which determines both learnability and sample complexity. Extending this to multiclass classification has long been challenging, since Natarajan's work in the late 80s proposing the Natarajan dimension (Nat) as a natural analogue of VC. Daniely and Shalev-Shwartz (2014) introduced the DS dimension, later shown by Brukhim et al. (2022) to characterize multiclass learnability. Brukhim et al. also showed that Nat and DS can diverge arbitrarily, suggesting that multiclass learning is governed by DS rather than Nat. We show that agnostic multiclass PAC sample complexity is in fact governed by two distinct dimensions. Specifically, we prove nearly tight agnostic sample complexity bounds that, up to log factors, take the form $\frac{DS^{1.5}}ε + \frac{Nat}{ε^2}$ where $ε$ is the excess risk. This bound is tight up to a $\sqrt{DS}$ factor in the first term, nearly matching known $Nat/ε^2$ and $DS/ε$ lower bounds. The first term reflects the DS-controlled regime, while the second shows that the Natarajan dimension still dictates asymptotic behavior for small $ε$. Thus, unlike binary or online classification -- where a single dimension (VC or Littlestone) controls both phenomena -- multiclass learning inherently involves two structural parameters. Our technical approach departs from traditional agnostic learning methods based on uniform convergence or reductions to realizable cases. A key ingredient is a novel online procedure based on a self-adaptive multiplicative-weights algorithm performing a label-space reduction, which may be of independent interest.

Sample Complexity of Agnostic Multiclass Classification: Natarajan Dimension Strikes Back

TL;DR

It is shown that agnostic multiclass PAC sample complexity is in fact governed by two distinct dimensions, and it is proved nearly tight agnostic sample complexity bounds that, up to log factors, take the form of where is the excess risk.

Abstract

The fundamental theorem of statistical learning states that binary PAC learning is governed by a single parameter -- the Vapnik-Chervonenkis (VC) dimension -- which determines both learnability and sample complexity. Extending this to multiclass classification has long been challenging, since Natarajan's work in the late 80s proposing the Natarajan dimension (Nat) as a natural analogue of VC. Daniely and Shalev-Shwartz (2014) introduced the DS dimension, later shown by Brukhim et al. (2022) to characterize multiclass learnability. Brukhim et al. also showed that Nat and DS can diverge arbitrarily, suggesting that multiclass learning is governed by DS rather than Nat. We show that agnostic multiclass PAC sample complexity is in fact governed by two distinct dimensions. Specifically, we prove nearly tight agnostic sample complexity bounds that, up to log factors, take the form where is the excess risk. This bound is tight up to a factor in the first term, nearly matching known and lower bounds. The first term reflects the DS-controlled regime, while the second shows that the Natarajan dimension still dictates asymptotic behavior for small . Thus, unlike binary or online classification -- where a single dimension (VC or Littlestone) controls both phenomena -- multiclass learning inherently involves two structural parameters. Our technical approach departs from traditional agnostic learning methods based on uniform convergence or reductions to realizable cases. A key ingredient is a novel online procedure based on a self-adaptive multiplicative-weights algorithm performing a label-space reduction, which may be of independent interest.

Paper Structure

This paper contains 27 sections, 14 theorems, 162 equations, 8 algorithms.

Table of Contents

  1. Introduction
  2. Main Result.
  3. Technical Overview
  4. Algorithm overview.
  5. Combining the three stages.
  6. Approximation by improper finite cover
  7. Learning a list
  8. Learning a list-bounded classifier
  9. Connections to other recent results
  10. Organization
  11. Preliminaries
  12. Notation
  13. Multiclass PAC learning
  14. Dimensions of hypothesis classes
  15. Sample compression scheme
  16. ...and 12 more sections

Key Result

Theorem 1.1

Let $\mathcal{H} \subseteq \mathcal{Y}^\mathcal{X}$ be a concept class, and let $\mathcal{M}_{\mathcal{H}}^{\mathrm{AG}}(\varepsilon,\delta)$ denote the sample complexity of agnostic PAC learning $\mathcal{H}$ with error $\varepsilon$ and confidence $1-\delta$. Then, where the $\widetilde{\Theta}$ notation hides logarithmic factors in $\mathtt{Nat}$, $d_{\mathrm{real}}$, and $1/\varepsilon$. A de

Theorems & Definitions (43)

  • Theorem 1.1: Main
  • Definition 2.1: Multiclass learner
  • Definition 2.2: Multiclass Agnostic PAC learning
  • Definition 2.3: Realizability
  • Definition 2.4: Multiclass Realizable PAC learning
  • Definition 2.5: Natarajan dimension, natarajan:89
  • Definition 2.6: Pseudo-cube, brukhim2022characterization
  • Definition 2.7: DS dimension, daniely2014optimal
  • Definition 2.8
  • Definition 2.9: Selection scheme
  • ...and 33 more