Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Adversarial Consistency of Surrogate Risks for Binary Classification

Natalie Frank, Jonathan Niles-Weed

TL;DR

This work provides a complete characterization of adversarially consistent surrogate losses for robust binary classification under $\epsilon$-ball perturbations. It identifies a precise necessary-and-sufficient condition, $C_\phi^*(\tfrac{1}{2}) < \phi(0)$, under which a surrogate is adversarially consistent, and demonstrates that common convex losses fail while the $\rho$-margin loss and shifted sigmoid satisfy. The authors leverage minimax duality with $W_\infty$ perturbation sets to relate adversarial risks to dual objectives, and they prove a quantitative excess-risk bound for the $\rho$-margin loss, showing that minimizing its adversarial surrogate risk effectively reduces the adversarial classification error. These results guide the design of surrogate losses for robust learning and lay groundwork for extending the theory to other perturbation models and loss families.

Abstract

We study the consistency of surrogate risks for robust binary classification. It is common to learn robust classifiers by adversarial training, which seeks to minimize the expected $0$-$1$ loss when each example can be maliciously corrupted within a small ball. We give a simple and complete characterization of the set of surrogate loss functions that are \emph{consistent}, i.e., that can replace the $0$-$1$ loss without affecting the minimizing sequences of the original adversarial risk, for any data distribution. We also prove a quantitative version of adversarial consistency for the $ρ$-margin loss. Our results reveal that the class of adversarially consistent surrogates is substantially smaller than in the standard setting, where many common surrogates are known to be consistent.

The Adversarial Consistency of Surrogate Risks for Binary Classification

TL;DR

This work provides a complete characterization of adversarially consistent surrogate losses for robust binary classification under -ball perturbations. It identifies a precise necessary-and-sufficient condition, , under which a surrogate is adversarially consistent, and demonstrates that common convex losses fail while the -margin loss and shifted sigmoid satisfy. The authors leverage minimax duality with perturbation sets to relate adversarial risks to dual objectives, and they prove a quantitative excess-risk bound for the -margin loss, showing that minimizing its adversarial surrogate risk effectively reduces the adversarial classification error. These results guide the design of surrogate losses for robust learning and lay groundwork for extending the theory to other perturbation models and loss families.

Abstract

We study the consistency of surrogate risks for robust binary classification. It is common to learn robust classifiers by adversarial training, which seeks to minimize the expected - loss when each example can be maliciously corrupted within a small ball. We give a simple and complete characterization of the set of surrogate loss functions that are \emph{consistent}, i.e., that can replace the - loss without affecting the minimizing sequences of the original adversarial risk, for any data distribution. We also prove a quantitative version of adversarial consistency for the -margin loss. Our results reveal that the class of adversarially consistent surrogates is substantially smaller than in the standard setting, where many common surrogates are known to be consistent.
Paper Structure (16 sections, 22 theorems, 60 equations)

This paper contains 16 sections, 22 theorems, 60 equations.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Setup
  4. Surrogate Risks
  5. Minimax Theorems for Adversarial Risks
  6. Adversarially Consistent Losses
  7. Approximate Complimentary Slackness
  8. Adversarial Consistency
  9. Quantitative Bounds for the $\rho$-Margin Loss
  10. Conclusion
  11. An Alternative Characterization of Consistency-- Proof of Proposition \ref{['prop:alt_consistency_characterization']}
  12. Minimizing $R_\phi^\epsilon$ over $\overline \mathbb{R}$-valued functions
  13. Further Properties of Adversarially Consistent Losses-- Proofs of Lemma \ref{['lemma:a_def_main']}, Lemma \ref{['lemma:minimizing_seq']}, and Proposition \ref{['prop:a_def_consistent']}
  14. Optimal Transport Facts--- Proof of Lemma \ref{['lemma:S_e_inequality']}
  15. Proof of Theorem \ref{['th:minimax_classification']}
  16. ...and 1 more sections

Key Result

Proposition 1

The following are equivalent:

Theorems & Definitions (39)

  • Proposition 1
  • Definition 1
  • Proposition 2
  • proof
  • Proposition 3
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Theorem 2
  • Proposition 4
  • ...and 29 more