Realizable Learning is All You Need
Max Hopkins, Daniel M. Kane, Shachar Lovett, Gaurav Mahajan
TL;DR
This work introduces a model-independent, three-line blackbox reduction that proves the realizable-to-agnostic learning equivalence holds across a wide range of learning paradigms beyond classical PAC, including distribution-family models, general losses, robustness, partiality, fairness, and privacy. Central to the approach is non-uniform covering: a randomized, finite-size cover of hypotheses derived from applying a realizable PAC-learner to all labelings of a unlabeled sample, which enables agnostic learning via empirical risk minimization over a small set. The authors also present four modification archetypes (discretization, subsampling, replacing the ERM, and changing the base model) to extend the reduction to infinite label spaces, malicious noise, semi-private learning, and covariate shift, respectively, yielding new bounds and insights in several settings (robust, private, SQ, etc.). Together, these results unify the understanding of realizable versus agnostic learnability, demonstrate the power of non-uniform covers over uniform ones, and provide practical reductions that improve or match known bounds in several regimes, including semi-private learning with optimal unlabeled sample complexity. The framework thus offers a wide, scalable pathway to transfer learnability guarantees across diverse models, suggesting a broad, underlying principle—property generalization—that captures the core essence of learnability across many contexts.
Abstract
The equivalence of realizable and agnostic learnability is a fundamental phenomenon in learning theory. With variants ranging from classical settings like PAC learning and regression to recent trends such as adversarially robust learning, it's surprising that we still lack a unified theory; traditional proofs of the equivalence tend to be disparate, and rely on strong model-specific assumptions like uniform convergence and sample compression. In this work, we give the first model-independent framework explaining the equivalence of realizable and agnostic learnability: a three-line blackbox reduction that simplifies, unifies, and extends our understanding across a wide variety of settings. This includes models with no known characterization of learnability such as learning with arbitrary distributional assumptions and more general loss functions, as well as a host of other popular settings such as robust learning, partial learning, fair learning, and the statistical query model. More generally, we argue that the equivalence of realizable and agnostic learning is actually a special case of a broader phenomenon we call property generalization: any desirable property of a learning algorithm (e.g. noise tolerance, privacy, stability) that can be satisfied over finite hypothesis classes extends (possibly in some variation) to any learnable hypothesis class.