Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uniform Computability of PAC Learning

Vasco Brattka, Guillaume Chirache

TL;DR

This work provides a uniform, Weihrauch-analytic classification of PAC learning under three information regimes (positive, negative, full) for closed concept classes, revealing that relative PAC learning aligns with Weak König's Lemma, proper PAC learning with finite VC-dimension collapses to limit-like computations, and improper PAC learning from positive information resides in the finitary DNC range. By studying VC-dimension computations and the VC-dimension operation, the authors connect learnability to precise Weihrauch degrees and Borel complexity, showing that different information representations and VC-dimension promises fundamentally change the constructivity of learning. The results yield a detailed map of when PAC learnability is computable, limit-computable, or non-deterministically computable, and illuminate the role of witnesses and VC-dimension in shaping uniform learnability. Overall, the work advances the understanding of the Fundamental Theorem of Statistical Learning from a uniform, computability-theoretic perspective and clarifies the boundaries of algorithmic learnability in a rigorous, degree-theoretic framework.

Abstract

We study uniform computability properties of PAC learning using Weihrauch complexity. We focus on closed concept classes, which are either represented by positive, by negative or by full information. Among other results, we prove that proper PAC learning from positive information is equivalent to the limit operation on Baire space, whereas improper PAC learning from positive information is closely related to Weak Kőnig's Lemma and even equivalent to it, when we have some negative information about the admissible hypotheses. If arbitrary hypotheses are allowed, then improper PAC learning from positive information is still in a finitary DNC range, which implies that it is non-deterministically computable, but does not allow for probabilistic algorithms. These results can also be seen as a classification of the degree of constructivity of the Fundamental Theorem of Statistical Learning. All the aforementioned results hold if an upper bound of the VC dimension is provided as an additional input information. We also study the question of how these results are affected if the VC dimension is not given, but only promised to be finite or if concept classes are represented by negative or full information. Finally, we also classify the complexity of the VC dimension operation itself, which is a problem that is of independent interest. For positive or full information it turns out to be equivalent to the binary sorting problem, for negative information it is equivalent to the jump of sorting. This classification allows also conclusions regarding the Borel complexity of PAC learnability.

Uniform Computability of PAC Learning

TL;DR

This work provides a uniform, Weihrauch-analytic classification of PAC learning under three information regimes (positive, negative, full) for closed concept classes, revealing that relative PAC learning aligns with Weak König's Lemma, proper PAC learning with finite VC-dimension collapses to limit-like computations, and improper PAC learning from positive information resides in the finitary DNC range. By studying VC-dimension computations and the VC-dimension operation, the authors connect learnability to precise Weihrauch degrees and Borel complexity, showing that different information representations and VC-dimension promises fundamentally change the constructivity of learning. The results yield a detailed map of when PAC learnability is computable, limit-computable, or non-deterministically computable, and illuminate the role of witnesses and VC-dimension in shaping uniform learnability. Overall, the work advances the understanding of the Fundamental Theorem of Statistical Learning from a uniform, computability-theoretic perspective and clarifies the boundaries of algorithmic learnability in a rigorous, degree-theoretic framework.

Abstract

We study uniform computability properties of PAC learning using Weihrauch complexity. We focus on closed concept classes, which are either represented by positive, by negative or by full information. Among other results, we prove that proper PAC learning from positive information is equivalent to the limit operation on Baire space, whereas improper PAC learning from positive information is closely related to Weak Kőnig's Lemma and even equivalent to it, when we have some negative information about the admissible hypotheses. If arbitrary hypotheses are allowed, then improper PAC learning from positive information is still in a finitary DNC range, which implies that it is non-deterministically computable, but does not allow for probabilistic algorithms. These results can also be seen as a classification of the degree of constructivity of the Fundamental Theorem of Statistical Learning. All the aforementioned results hold if an upper bound of the VC dimension is provided as an additional input information. We also study the question of how these results are affected if the VC dimension is not given, but only promised to be finite or if concept classes are represented by negative or full information. Finally, we also classify the complexity of the VC dimension operation itself, which is a problem that is of independent interest. For positive or full information it turns out to be equivalent to the binary sorting problem, for negative information it is equivalent to the jump of sorting. This classification allows also conclusions regarding the Borel complexity of PAC learnability.
Paper Structure (7 sections, 60 theorems, 61 equations, 6 figures)

This paper contains 7 sections, 60 theorems, 61 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Preliminaries on Computable Analysis and Weihrauch Complexity
  3. VC Dimension
  4. Relative PAC Learning
  5. Proper PAC Learning
  6. Improper PAC Learning
  7. Conclusions

Key Result

Theorem 3

Let $\varnothing\not={\mathcal{C}}\subseteq2^{\mathbb{N}}$. Then In fact, one can make the following additional quantitative statementsThe bounds given in this theorem are typically expressed in terms of $\varepsilon=2^{-i}$ and $\delta=2^{-j}$, in which case they read m_d(\varepsilon,\delta)=c\cdot\frac{d\cdot\log(\frac{1}{\varepsilon})+\log(\frac{1}{\delta})}{\

Figures (6)

  • Figure 1: Computability properties of $\mathsf{PPAC},\mathsf{PPAC}_{<\infty}$ and $\mathsf{IPAC},\mathsf{IPAC}_{<\infty}$, respectively.
  • Figure 2: PAC learning in the strong Weihrauch lattice.
  • Figure 3: Weihrauch reducibility $f\mathop{\leq_{\mathrm{W}}} g$.
  • Figure 4: The vicinity of $\mathsf{DNC}$ in the Weihrauch lattice.
  • Figure 5: Costs of PAC learning for closed concept classes ${\mathcal{C}}\subseteq2^{\mathbb{N}}$ of finite VC dimension.
  • ...and 1 more figures

Theorems & Definitions (96)

  • Definition 1: PAC learning
  • Definition 2: PAC learnability
  • Theorem 3: Fundamental Theorem of Statistical Learning
  • Definition 4: PAC learning problems
  • Definition 5: Bounded PAC learning problems
  • Definition 6: Witnesses
  • Theorem 7: Delle Rose, Kozachinskiy, Rojas, and Steifer 2023
  • Definition 8: Hyperspace representations
  • Proposition 9: Sequential representation of closed subsets
  • Proposition 10: Pruned trees
  • ...and 86 more