Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Computability of Robust PAC Learning

Pascale Gourdeau, Tosca Lechner, Ruth Urner

TL;DR

This work initiates the study of computability constraints in adversarially robust PAC learning (robust CPAC) and develops a formal framework to analyze how computability on learners and perturbations interacts with robustness. It provides simple sufficient conditions for robust CPAC learnability while also ranking the limits by presenting impossibility results showing that CPAC learnability and robust learnability do not in general imply robust CPAC learnability; it also reveals surprising facets, such as robust CPAC learnability not requiring computable robust losses. A key contribution is the computable robust shattering dimension, shown to be a necessary but not sufficient condition for robust CPAC learnability, along with a computable no-free-lunch theorem that highlights fundamental limits in the robust setting. Together, these results connect computability theory with robust learning, offering new conceptual tools and pointing to directions for future work on robustness under computational constraints and the design of robust learning algorithms under practical computability limits.

Abstract

We initiate the study of computability requirements for adversarially robust learning. Adversarially robust PAC-type learnability is by now an established field of research. However, the effects of computability requirements in PAC-type frameworks are only just starting to emerge. We introduce the problem of robust computable PAC (robust CPAC) learning and provide some simple sufficient conditions for this. We then show that learnability in this setup is not implied by the combination of its components: classes that are both CPAC and robustly PAC learnable are not necessarily robustly CPAC learnable. Furthermore, we show that the novel framework exhibits some surprising effects: for robust CPAC learnability it is not required that the robust loss is computably evaluable! Towards understanding characterizing properties, we introduce a novel dimension, the computable robust shattering dimension. We prove that its finiteness is necessary, but not sufficient for robust CPAC learnability. This might yield novel insights for the corresponding phenomenon in the context of robust PAC learnability, where insufficiency of the robust shattering dimension for learnability has been conjectured, but so far a resolution has remained elusive.

On the Computability of Robust PAC Learning

TL;DR

This work initiates the study of computability constraints in adversarially robust PAC learning (robust CPAC) and develops a formal framework to analyze how computability on learners and perturbations interacts with robustness. It provides simple sufficient conditions for robust CPAC learnability while also ranking the limits by presenting impossibility results showing that CPAC learnability and robust learnability do not in general imply robust CPAC learnability; it also reveals surprising facets, such as robust CPAC learnability not requiring computable robust losses. A key contribution is the computable robust shattering dimension, shown to be a necessary but not sufficient condition for robust CPAC learnability, along with a computable no-free-lunch theorem that highlights fundamental limits in the robust setting. Together, these results connect computability theory with robust learning, offering new conceptual tools and pointing to directions for future work on robustness under computational constraints and the design of robust learning algorithms under practical computability limits.

Abstract

We initiate the study of computability requirements for adversarially robust learning. Adversarially robust PAC-type learnability is by now an established field of research. However, the effects of computability requirements in PAC-type frameworks are only just starting to emerge. We introduce the problem of robust computable PAC (robust CPAC) learning and provide some simple sufficient conditions for this. We then show that learnability in this setup is not implied by the combination of its components: classes that are both CPAC and robustly PAC learnable are not necessarily robustly CPAC learnable. Furthermore, we show that the novel framework exhibits some surprising effects: for robust CPAC learnability it is not required that the robust loss is computably evaluable! Towards understanding characterizing properties, we introduce a novel dimension, the computable robust shattering dimension. We prove that its finiteness is necessary, but not sufficient for robust CPAC learnability. This might yield novel insights for the corresponding phenomenon in the context of robust PAC learnability, where insufficiency of the robust shattering dimension for learnability has been conjectured, but so far a resolution has remained elusive.
Paper Structure (36 sections, 13 theorems, 41 equations, 3 algorithms)

This paper contains 36 sections, 13 theorems, 41 equations, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Computable Learning.
  4. Robust Learning.
  5. Problem Set-up
  6. Notation.
  7. Computability.
  8. Learnability.
  9. Computable learnability.
  10. Adversarial robustness.
  11. Computable robust learnability.
  12. Proper learnability and empirical risk minimization.
  13. Useful facts.
  14. Warm-up: Simple Sufficient Conditions for Robust CPAC Learnability
  15. Relating CPAC and Robust CPAC Learning
  16. ...and 21 more sections

Key Result

Proposition 14

Let $\mathcal{H}$ be computably online learnable, and let $\mathcal{U}$ be such that $\mathsf{VC}(\mathcal{H}_\text{mar}^\mathcal{U})<\infty$. Then $\mathcal{H}$ is $\mathcal{U}$-robustly CPAC learnable with access to the PAO in the realizable setting.

Theorems & Definitions (45)

  • Definition 1: Agnostic PAC Learnability
  • Definition 2: Shattering and VC dimension vapnik1971uniform
  • Definition 3: Computable Representation of a Hypothesis class agarwal2020learnability
  • Definition 4: CPAC Learnability, agarwal2020learnability
  • Definition 5: Agnostic Robust PAC Learnability montasser2019vc
  • Definition 6: Robust CPAC Learnability
  • Definition 7: Representations of Perturbation Types
  • Definition 8: Robust loss computably evaluable
  • proof
  • proof
  • ...and 35 more