Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Characterization of Semi-Supervised Adversarially-Robust PAC Learnability

Idan Attias, Steve Hanneke, Yishay Mansour

TL;DR

This work provides a comprehensive theory for semi-supervised adversarially-robust PAC learning under distribution-free assumptions, introducing the central complexity measures $VC_{\mathcal{U}}(\mathcal{H})$ and $RS_{\mathcal{U}}(\mathcal{H})$ to quantify labeled and unlabeled data requirements. It presents the GRASS algorithm, which leverages partial concept learning on a robustly reduced class and subsequent robust agnostic learning on labeled unlabeled data, achieving near-optimal labeled sample complexity $\tilde{O}\left( \frac{VC_{\mathcal{U}}(\mathcal{H})}{\epsilon} \right)$ and unlabeled complexity tied to $\Lambda_{AG}$, thereby separating semi-supervised from supervised label requirements. The paper establishes tight realizable and agnostic bounds, demonstrates a fundamental gap between semi-supervised and supervised label complexities that does not occur in non-robust PAC learning, and shows the necessity of improper learning in general settings. These results highlight a significant practical advantage of semi-supervised approaches for robust learning and open questions about optimal rates in the agnostic regime with limited labeled data.

Abstract

We study the problem of learning an adversarially robust predictor to test time attacks in the semi-supervised PAC model. We address the question of how many labeled and unlabeled examples are required to ensure learning. We show that having enough unlabeled data (the size of a labeled sample that a fully-supervised method would require), the labeled sample complexity can be arbitrarily smaller compared to previous works, and is sharply characterized by a different complexity measure. We prove nearly matching upper and lower bounds on this sample complexity. This shows that there is a significant benefit in semi-supervised robust learning even in the worst-case distribution-free model, and establishes a gap between the supervised and semi-supervised label complexities which is known not to hold in standard non-robust PAC learning.

A Characterization of Semi-Supervised Adversarially-Robust PAC Learnability

TL;DR

This work provides a comprehensive theory for semi-supervised adversarially-robust PAC learning under distribution-free assumptions, introducing the central complexity measures and to quantify labeled and unlabeled data requirements. It presents the GRASS algorithm, which leverages partial concept learning on a robustly reduced class and subsequent robust agnostic learning on labeled unlabeled data, achieving near-optimal labeled sample complexity and unlabeled complexity tied to , thereby separating semi-supervised from supervised label requirements. The paper establishes tight realizable and agnostic bounds, demonstrates a fundamental gap between semi-supervised and supervised label complexities that does not occur in non-robust PAC learning, and shows the necessity of improper learning in general settings. These results highlight a significant practical advantage of semi-supervised approaches for robust learning and open questions about optimal rates in the agnostic regime with limited labeled data.

Abstract

We study the problem of learning an adversarially robust predictor to test time attacks in the semi-supervised PAC model. We address the question of how many labeled and unlabeled examples are required to ensure learning. We show that having enough unlabeled data (the size of a labeled sample that a fully-supervised method would require), the labeled sample complexity can be arbitrarily smaller compared to previous works, and is sharply characterized by a different complexity measure. We prove nearly matching upper and lower bounds on this sample complexity. This shows that there is a significant benefit in semi-supervised robust learning even in the worst-case distribution-free model, and establishes a gap between the supervised and semi-supervised label complexities which is known not to hold in standard non-robust PAC learning.
Paper Structure (28 sections, 18 theorems, 34 equations, 1 algorithm)

This paper contains 28 sections, 18 theorems, 34 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background.
  3. Main contributions.
  4. Related work.
  5. Preliminaries
  6. Realizable robust $\operatorname{PAC}$ learning.
  7. Agnostic robust $\operatorname{PAC}$ learning.
  8. Partial concept classes alon2021theory.
  9. Warm-up: knowing the support of the marginal distribution
  10. Near-optimal semi-supervised sample complexity
  11. Algorithm motivation.
  12. Algorithm outline and guarantees.
  13. Proper vs. improper.
  14. Agnostic robust learning
  15. Open question.
  16. ...and 13 more sections

Key Result

theorem 3.1

For hypothesis class $\mathcal{H}$ and adversary $\mathcal{U}$, when the support of the marginal distribution $\mathcal{D}_{\mathcal{X}}$ is known, the labeled sample complexity is $\Theta\left( \frac{\operatorname{VC_{\mathcal{U}}}(\mathcal{H})}{\epsilon}+\frac{\log\frac{1}{\delta}}{\epsilon} \righ

Theorems & Definitions (39)

  • Definition 1.1: $\operatorname{VC_{\mathcal{U}}}$-dimension
  • Definition 1.2: $\mathrm{RS}_\mathcal{U}$-dimension
  • Definition 2.1: Realizable robust $\operatorname{PAC}$ learnability
  • Definition 2.2: Realizable semi-supervised robust $\operatorname{PAC}$ learnability
  • Definition 2.3: Agnostic robust $\operatorname{PAC}$ learnability
  • Definition 2.4: Agnostic semi-supervised robust $\operatorname{PAC}$ learnability
  • theorem 3.1
  • Proposition 3.2: montasser2019vc, Proposition 9
  • Corollary 3.3
  • Definition 4.1
  • ...and 29 more