Simplifying Adversarially Robust PAC Learning with Tolerance
Hassan Ashtiani, Vinayak Pathak, Ruth Urner
TL;DR
The paper tackles adversarially robust PAC learning by introducing a tolerance-based relaxation that allows learning with a target perturbation set V while aiming for robustness against a smaller set U; this enables simple, PAC-style guarantees with a sample complexity that scales linearly in the VC-dimension. The authors propose a two-stage supervised tolerant learning approach that first performs Robust ERM on V and then applies a smoothing step to produce a predictor that is almost from the original hypothesis class H, achieving practical, near-proper results. They provide realizable and agnostic supervised algorithms, including a global discretization variant that avoids intricate subroutines and yields tight bounds, as well as semi-supervised tolerant learners that leverage unlabeled data to match prior bounds with simpler procedures. A supporting lower bound shows that some degree of impropriety is unavoidable, underscoring the value of the tolerant framework. Overall, tolerance substantially simplifies robust learning, enabling PAC-type guarantees and practical semi-supervised extensions for adversarially robust classification.
Abstract
Adversarially robust PAC learning has proved to be challenging, with the currently best known learners [Montasser et al., 2021a] relying on improper methods based on intricate compression schemes, resulting in sample complexity exponential in the VC-dimension. A series of follow up work considered a slightly relaxed version of the problem called adversarially robust learning with tolerance [Ashtiani et al., 2023, Bhattacharjee et al., 2023, Raman et al., 2024] and achieved better sample complexity in terms of the VC-dimension. However, those algorithms were either improper and complex, or required additional assumptions on the hypothesis class H. We prove, for the first time, the existence of a simpler learner that achieves a sample complexity linear in the VC-dimension without requiring additional assumptions on H. Even though our learner is improper, it is "almost proper" in the sense that it outputs a hypothesis that is "similar" to a hypothesis in H. We also use the ideas from our algorithm to construct a semi-supervised learner in the tolerant setting. This simple algorithm achieves comparable bounds to the previous (non-tolerant) semi-supervised algorithm of Attias et al. [2022a], but avoids the use of intricate subroutines from previous works, and is "almost proper."