Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Probably Approximately Correct Learning Model in Computational Learning Theory

Rocco A. Servedio

TL;DR

Valiant's PAC learning framework formalizes when efficient learning is possible by balancing statistical accuracy and computational tractability under arbitrary data distributions. The survey organizes the theory around the distribution-free model, its variants, and corresponding algorithms, including the elimination approach, feature expansion, and linear-programming methods that yield poly-time learning for several Boolean function classes, such as conjunctions, k-CNFs, and DNFs, while also detailing the hardness landscape. It covers robustness to noise via malicious, agnostic, and random classification noise models, and discusses the impact of membership queries through the PAC$+$MQ and related exact-learning connections. Finally, it surveys representation-dependent and -independent hardness results rooted in cryptography and average-case complexity, illustrating fundamental limits that persist even under favorable modeling assumptions, and it emphasizes the interdisciplinary influence of PAC ideas on boosting, SQ-learning, privacy, and beyond.

Abstract

This survey paper gives an overview of various known results on learning classes of Boolean functions in Valiant's Probably Approximately Correct (PAC) learning model and its commonly studied variants.

The Probably Approximately Correct Learning Model in Computational Learning Theory

TL;DR

Valiant's PAC learning framework formalizes when efficient learning is possible by balancing statistical accuracy and computational tractability under arbitrary data distributions. The survey organizes the theory around the distribution-free model, its variants, and corresponding algorithms, including the elimination approach, feature expansion, and linear-programming methods that yield poly-time learning for several Boolean function classes, such as conjunctions, k-CNFs, and DNFs, while also detailing the hardness landscape. It covers robustness to noise via malicious, agnostic, and random classification noise models, and discusses the impact of membership queries through the PACMQ and related exact-learning connections. Finally, it surveys representation-dependent and -independent hardness results rooted in cryptography and average-case complexity, illustrating fundamental limits that persist even under favorable modeling assumptions, and it emphasizes the interdisciplinary influence of PAC ideas on boosting, SQ-learning, privacy, and beyond.

Abstract

This survey paper gives an overview of various known results on learning classes of Boolean functions in Valiant's Probably Approximately Correct (PAC) learning model and its commonly studied variants.

Paper Structure

This paper contains 19 sections, 22 theorems, 7 equations, 4 figures.

Table of Contents

  1. Introduction
  2. The PAC Learning Framework
  3. Motivation and intuition.
  4. The distribution-free PAC learning model and some variants.
  5. Key ingredients of the PAC framework.
  6. An example of distribution-free PAC learning: Learning Boolean conjunctions.
  7. The PAC learnability research program.
  8. Positive results: PAC Learning Algorithms
  9. Distribution-free PAC learning from random examples.
  10. Distribution-free PAC learning with membership queries: the "PAC $+$ MQ" model
  11. PAC learning in the presence of noise.
  12. Distribution-specific PAC learning
  13. Hardness of PAC learning
  14. Representation-dependent hardness
  15. Representation-independent hardness
  16. ...and 4 more sections

Key Result

Theorem 1

The class ${\cal C}$ of Boolean conjunctions is distribution-free PAC learnable in time $\mathrm{poly}(n,1/\varepsilon,\log(1/\delta))$ using $\mathrm{poly}(n,1/\varepsilon,\log(1/\delta))$ samples.

Figures (4)

  • Figure 1: A 1-decision list "if $\overline{x}_3$ then output 1 else if $\overline{x}_2$ then output $-1$ else if $x_7$ then output $-1$ else output $1$."
  • Figure 2: A decision tree of size eight and rank two.
  • Figure 3: A collection of two-dimensional data points that have been labeled according to an intersection of three LTFs (the decision boundaries for the LTFs are depicted using dashed lines).
  • Figure 4: An input graph $G$ with $n=5$ nodes and $m=6$ edges, and the output $S^+ \cup S^-$ of the transformation $T$.

Theorems & Definitions (23)

  • Claim 2.1: BEH+:87
  • Theorem 1
  • Theorem 2: Valiant:84PittValiant:86
  • Theorem 3: Rivest:87
  • Theorem 4: EhrenfeuchtHaussler:89
  • Theorem 5: FischerSimon:92HSW:92
  • Theorem 6: BEH+:89
  • Theorem 7: BEH+:89
  • Theorem 8: KS04
  • Theorem 9: Lee09formulas
  • ...and 13 more