Do PAC-Learners Learn the Marginal Distribution?
Max Hopkins, Daniel M. Kane, Shachar Lovett, Gaurav Mahajan
TL;DR
This work revisits PAC learning under distribution-family constraints, moving beyond the traditional distribution-free paradigm. It introduces two refined density-estimation notions—intermediate density estimation (IDE) and weak density estimation (WDE)—and shows a finer PAC-density relationship: WDE ⊊ PAC-Learning ⊊ IDE. Under reasonable assumptions such as uniformly bounded metric entropy (UBME), density estimation becomes equivalent to uniform estimation (DE ⇔ UE), clarifying how the classical Fundamental Theorem can extend beyond distribution-free settings. Together, the results provide a more nuanced understanding of learnability under distributional restrictions and illuminate the limits and possibilities of extending PAC theory to structured distribution families.
Abstract
The Fundamental Theorem of PAC Learning asserts that learnability of a concept class $H$ is equivalent to the $\textit{uniform convergence}$ of empirical error in $H$ to its mean, or equivalently, to the problem of $\textit{density estimation}$, learnability of the underlying marginal distribution with respect to events in $H$. This seminal equivalence relies strongly on PAC learning's `distribution-free' assumption, that the adversary may choose any marginal distribution over data. Unfortunately, the distribution-free model is known to be overly adversarial in practice, failing to predict the success of modern machine learning algorithms, but without the Fundamental Theorem our theoretical understanding of learning under distributional constraints remains highly limited. In this work, we revisit the connection between PAC learning, uniform convergence, and density estimation beyond the distribution-free setting when the adversary is restricted to choosing a marginal distribution from a known family $\mathscr{P}$. We prove that while the traditional Fundamental Theorem indeed fails, a finer-grained connection between the three fundamental notions continues to hold: 1. PAC-Learning is strictly sandwiched between two refined models of density estimation, differing only in whether the learner $\textit{knows}$ the set of well-estimated events in $H$. 2. Under reasonable assumptions on $H$ and $\mathscr{P}$, density estimation is equivalent to $\textit{uniform estimation}$, a relaxation of uniform convergence allowing non-empirical estimators. Together, our results give a clearer picture of how the Fundamental Theorem extends beyond the distribution-free setting and shed new light on the classically challenging problem of learning under arbitrary distributional assumptions.