Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust optimization for adversarial learning with finite sample complexity guarantees

André Bertolace, Konstatinos Gatsis, Kostas Margellos

TL;DR

This paper introduces a novel adversarial training method for robust linear and nonlinear classifiers, inspired by Support Vector Machine (SVM) margins, and derives finite sample complexity bounds for binary and multi-class classifiers, which align with those of natural classifiers.

Abstract

Decision making and learning in the presence of uncertainty has attracted significant attention in view of the increasing need to achieve robust and reliable operations. In the case where uncertainty stems from the presence of adversarial attacks this need is becoming more prominent. In this paper we focus on linear and nonlinear classification problems and propose a novel adversarial training method for robust classifiers, inspired by Support Vector Machine (SVM) margins. We view robustness under a data driven lens, and derive finite sample complexity bounds for both linear and non-linear classifiers in binary and multi-class scenarios. Notably, our bounds match natural classifiers' complexity. Our algorithm minimizes a worst-case surrogate loss using Linear Programming (LP) and Second Order Cone Programming (SOCP) for linear and non-linear models. Numerical experiments on the benchmark MNIST and CIFAR10 datasets show our approach's comparable performance to state-of-the-art methods, without needing adversarial examples during training. Our work offers a comprehensive framework for enhancing binary linear and non-linear classifier robustness, embedding robustness in learning under the presence of adversaries.

Robust optimization for adversarial learning with finite sample complexity guarantees

TL;DR

This paper introduces a novel adversarial training method for robust linear and nonlinear classifiers, inspired by Support Vector Machine (SVM) margins, and derives finite sample complexity bounds for binary and multi-class classifiers, which align with those of natural classifiers.

Abstract

Decision making and learning in the presence of uncertainty has attracted significant attention in view of the increasing need to achieve robust and reliable operations. In the case where uncertainty stems from the presence of adversarial attacks this need is becoming more prominent. In this paper we focus on linear and nonlinear classification problems and propose a novel adversarial training method for robust classifiers, inspired by Support Vector Machine (SVM) margins. We view robustness under a data driven lens, and derive finite sample complexity bounds for both linear and non-linear classifiers in binary and multi-class scenarios. Notably, our bounds match natural classifiers' complexity. Our algorithm minimizes a worst-case surrogate loss using Linear Programming (LP) and Second Order Cone Programming (SOCP) for linear and non-linear models. Numerical experiments on the benchmark MNIST and CIFAR10 datasets show our approach's comparable performance to state-of-the-art methods, without needing adversarial examples during training. Our work offers a comprehensive framework for enhancing binary linear and non-linear classifier robustness, embedding robustness in learning under the presence of adversaries.
Paper Structure (42 sections, 12 theorems, 91 equations, 6 figures, 1 table)

This paper contains 42 sections, 12 theorems, 91 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our methodology and contribution
  3. Related work
  4. Learning in the presence of an adversary
  5. Adversarial attacks and related approaches
  6. Proposed approach: margin-inspired adversarial training
  7. Main results
  8. Sample complexity bounds
  9. Binary classifiers
  10. Linear binary classifiers with bounded inputs
  11. Kernel-based non-linear binary classifiers
  12. Multi-class classifiers
  13. Classifier computation
  14. Linear programming formulation for linear binary classifiers
  15. Second order cone programming formulation for kernel-based binary classifiers
  16. ...and 27 more sections

Key Result

Theorem III.1

Binary classifier. Consider the hypothesis class $\mathcal{H}$ of Lipschitz continuous functions. Fix any $\zeta>0$. We then have that, with probability at least $1-\delta$, for any $h \in \mathcal{H}$, The aforementioned bound holds uniformly, i.e., for $\gamma > 1$ and for any fixed $r>0$, with probability at least $1-\delta$, for all $\zeta \in ]0, r]$, and for any $h \in \mathcal{H}$,

Figures (6)

  • Figure 1: Graphical representation of the decision boundary and errors: natural error (blue), boundary error (dashed-red), and robust error (dashed-black).
  • Figure 2: Accuracy of linear classifiers using out-of-sample adversarial tampered data considering non-adversarial training, FGSM Goodfellow2015, PGD Madry2019, TRADES [$\lambda=1.0$] TRADES_ZhangElGhaoui2019 and proposed margin-based approach. Datasets: (a) NIST 0/1, (b) NIST 3/8, (c) CIFAR10 Airplane/Dog and (d) CIFAR10 Cat/Dog.
  • Figure 3: NIST 3: clean (left), tampered $\xi=0.1$ (middle) and tampered $\xi=0.25$ (right)
  • Figure 4: NIST 8: clean (left), tampered $\xi=0.1$ (middle) and tampered $\xi=0.25$ (right)
  • Figure 5: CIFAR10 cat: clean (left), tampered $\xi=0.1$ (middle) and tampered $\xi=0.25$ (right)
  • ...and 1 more figures

Theorems & Definitions (25)

  • Theorem III.1
  • proof
  • Remark III.1
  • Remark III.2
  • Remark III.3
  • Lemma III.1
  • proof
  • Corollary III.1
  • proof
  • Definition III.1
  • ...and 15 more