Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarially Robust PAC Learnability of Real-Valued Functions

Idan Attias, Steve Hanneke

TL;DR

This work advances the theory of adversarially robust PAC learning for real-valued functions under arbitrary perturbation sets. It shows that finite fat-shattering dimension suffices for robust learnability in both realizable and agnostic regimes, with convex classes allowing proper learning. The authors introduce novel adversarially robust sample compression schemes and an agnostic real-valued compression technique, and they significantly improve sample complexity in the $\ell_1$-style setting via median boosting and sparsification. The results extend robust learning from binary-valued concepts to regression, offering rigorous guarantees and new methodological tools for future exploration of robust real-valued prediction under perturbations.

Abstract

We study robustness to test-time adversarial attacks in the regression setting with $\ell_p$ losses and arbitrary perturbation sets. We address the question of which function classes are PAC learnable in this setting. We show that classes of finite fat-shattering dimension are learnable in both realizable and agnostic settings. Moreover, for convex function classes, they are even properly learnable. In contrast, some non-convex function classes provably require improper learning algorithms. Our main technique is based on a construction of an adversarially robust sample compression scheme of a size determined by the fat-shattering dimension. Along the way, we introduce a novel agnostic sample compression scheme for real-valued functions, which may be of independent interest.

Adversarially Robust PAC Learnability of Real-Valued Functions

TL;DR

This work advances the theory of adversarially robust PAC learning for real-valued functions under arbitrary perturbation sets. It shows that finite fat-shattering dimension suffices for robust learnability in both realizable and agnostic regimes, with convex classes allowing proper learning. The authors introduce novel adversarially robust sample compression schemes and an agnostic real-valued compression technique, and they significantly improve sample complexity in the -style setting via median boosting and sparsification. The results extend robust learning from binary-valued concepts to regression, offering rigorous guarantees and new methodological tools for future exploration of robust real-valued prediction under perturbations.

Abstract

We study robustness to test-time adversarial attacks in the regression setting with losses and arbitrary perturbation sets. We address the question of which function classes are PAC learnable in this setting. We show that classes of finite fat-shattering dimension are learnable in both realizable and agnostic settings. Moreover, for convex function classes, they are even properly learnable. In contrast, some non-convex function classes provably require improper learning algorithms. Our main technique is based on a construction of an adversarially robust sample compression scheme of a size determined by the fat-shattering dimension. Along the way, we introduce a novel agnostic sample compression scheme for real-valued functions, which may be of independent interest.
Paper Structure (31 sections, 10 theorems, 93 equations, 3 algorithms)

This paper contains 31 sections, 10 theorems, 93 equations, 3 algorithms.

Table of Contents

  1. Introduction
  2. Main results.
  3. Technical contributions and related work.
  4. Problem Setup and Preliminaries
  5. Learning models.
  6. Oracles.
  7. Complexity measures.
  8. Approximate sample compression schemes.
  9. Notation.
  10. Robust Regression
  11. Proof overview and algorithm outline.
  12. Improved Sample Complexity via Median Boosting and Sparsification
  13. Proof overview and algorithm outline.
  14. Robust $(\eta,\beta)$-Regression
  15. Agnostic setting
  16. ...and 16 more sections

Key Result

theorem 3.1

alg:lp-regression-highvote implies that the sample complexity for robust $(\epsilon,\delta)$-PAC learning a concept class $\mathcal{H}$ with the $\ell_p$ robust loss is for some numerical constant $c\in (0,\infty)$. Recall that $\mathrm{fat}^*\mathopen{}\left(\mathcal{F},\epsilon\right) \lesssim \frac{1}{\epsilon}2^{\mathrm{fat}\mathopen{}\left(\mathcal{F},\epsilon/2\right)+1}$ by eq:dual-fat.

Theorems & Definitions (21)

  • Definition 2.1: Robust regression
  • Definition 2.2: Robust $(\eta,\beta)$-regression
  • theorem 3.1
  • Remark 3.2
  • Remark 3.3
  • theorem 3.4: Generalization from approximate interpolation with changing cutoffs
  • theorem 4.1
  • Definition 4.2: Weak real-valued learner
  • theorem 5.1
  • theorem 5.2
  • ...and 11 more