Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Surrogate Risk Bounds for Binary Classification

Natalie S. Frank

TL;DR

This work investigates how surrogate losses used in adversarial binary classification control the adversarial classification risk $R^epsilon$, providing rate-based guarantees beyond mere adversarial consistency. It develops a linear surrogate risk bound under Massart-type noise conditions on the distribution of optimal adversarial attacks and a distribution-dependent convex-bound that applies when the surrogate is adversarially consistent, including for convex losses. The authors also derive bounds in non-consistent regimes by leveraging transport-map structures to localize the analysis and obtain concave, distribution-dependent guarantees. The results are underpinned by a minimax duality framework with $W_\infty$ perturbations and dual representations $\bar{R}$ and $\bar{R}_\u0015phi$, connecting optimal adversarial attacks, adversarial Bayes classifiers, and surrogate risks. Overall, the paper advances the theoretical understanding of how surrogate risk minimization translates into adversarial robustness, with implications for the design and analysis of robust learning methods.

Abstract

A central concern in classification is the vulnerability of machine learning models to adversarial attacks. Adversarial training is one of the most popular techniques for training robust classifiers, which involves minimizing an adversarial surrogate risk. Recent work has characterized the conditions under which any sequence minimizing the adversarial surrogate risk also minimizes the adversarial classification risk in the binary setting, a property known as adversarial consistency. However, these results do not address the rate at which the adversarial classification risk approaches its optimal value along such a sequence. This paper provides surrogate risk bounds that quantify that convergence rate.

Adversarial Surrogate Risk Bounds for Binary Classification

TL;DR

This work investigates how surrogate losses used in adversarial binary classification control the adversarial classification risk , providing rate-based guarantees beyond mere adversarial consistency. It develops a linear surrogate risk bound under Massart-type noise conditions on the distribution of optimal adversarial attacks and a distribution-dependent convex-bound that applies when the surrogate is adversarially consistent, including for convex losses. The authors also derive bounds in non-consistent regimes by leveraging transport-map structures to localize the analysis and obtain concave, distribution-dependent guarantees. The results are underpinned by a minimax duality framework with perturbations and dual representations and , connecting optimal adversarial attacks, adversarial Bayes classifiers, and surrogate risks. Overall, the paper advances the theoretical understanding of how surrogate risk minimization translates into adversarial robustness, with implications for the design and analysis of robust learning methods.

Abstract

A central concern in classification is the vulnerability of machine learning models to adversarial attacks. Adversarial training is one of the most popular techniques for training robust classifiers, which involves minimizing an adversarial surrogate risk. Recent work has characterized the conditions under which any sequence minimizing the adversarial surrogate risk also minimizes the adversarial classification risk in the binary setting, a property known as adversarial consistency. However, these results do not address the rate at which the adversarial classification risk approaches its optimal value along such a sequence. This paper provides surrogate risk bounds that quantify that convergence rate.

Paper Structure

This paper contains 39 sections, 44 theorems, 174 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Surrogate Risks
  4. Surrogate Risk Bounds
  5. Adversarial Risks
  6. Minimax Theorems
  7. Main Results
  8. Proof of Linear Surrogate Bounds
  9. Proof of \ref{['thm:adv_surrogate_massart']}
  10. Proof of \ref{['lemma:linear_adv_target']}
  11. Lower Bounds
  12. Proof of \ref{['thm:convex_surrogate_bound']}
  13. Proof of \ref{['thm:non_consistent_bound_linear', 'thm:non_consistent_bound_nonlinear']}
  14. Related Works
  15. Surrogate Risk Bounds:
  16. ...and 24 more sections

Key Result

Theorem 1

A loss $\phi$ is consistent iff $C_\phi^*(\eta)<\phi(0)$ for all $\eta\neq 1/2$.

Figures (3)

  • Figure 1: Adversarial Bayes classifiers for the distribution where ${\mathbb P}_0={\mathbb P}_1$ are uniform distributions on $\overline{B_\epsilon({\mathbf{0}})}$, the counterexample from MeunierEttedguietal22. The classifiers in (a) and (b) are equivalent up to degeneracy, as are those in (c) and (d), but the classifiers in (a) and (c) are not. A sequence minimizing $R_\phi^\epsilon$ but not $R^\epsilon$ is provided in \ref{['eq:f_n']}.
  • Figure 2: Ambiguous images in the MNIST and CIFAR10 datasets. (a) lies between a '5' and a '3', while (b) is difficult to classify at all, despite being labeled as a '5'. In CIFAR10, image (c) is ambiguous between a ship and an airplane, and image (d) is similarly hard to identify.
  • Figure 3: Distributions from \ref{['ex:realizable', 'ex:massart', 'ex:gaussian']} along with attacks that maximize the dual $\bar{R}_\phi$.

Theorems & Definitions (84)

  • Definition 1
  • Theorem 1
  • Theorem 2: TewariBartlett2007
  • Proposition 1
  • Definition 2
  • Theorem 3: FrankNilesWeed23consistency
  • Definition 3
  • Theorem 4: frank2024consistency
  • Lemma 1
  • Theorem 5: frank2024consistency
  • ...and 74 more