Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Risk Bounds via Function Transformation

Justin Khim, Po-Ling Loh

TL;DR

This work presents a principled framework for bounding adversarial risk by transforming the loss-augmented predictor into transformed predictors whose standard generalization bounds apply. The supremum transform handles linear classifiers exactly, yielding explicit bounds that separate optimization error, data geometry, and perturbation strength, while the tree transform extends this idea to neural networks, enabling finite-sample risk bounds via Rademacher complexity. The authors extend the theory to multiclass classification and regression, and propose optimization schemes to minimize these adversarial-risk bounds in practical settings. The results illuminate how adversarial perturbations affect generalization and connect robustness to distributional-robustness concepts, with practical algorithms for robust linear and neural-network training. Overall, the paper provides a versatile toolkit for analyzing and improving adversarial robustness under standard learning-theoretic guarantees.

Abstract

We derive bounds for a notion of adversarial risk, designed to characterize the robustness of linear and neural network classifiers to adversarial perturbations. Specifically, we introduce a new class of function transformations with the property that the risk of the transformed functions upper-bounds the adversarial risk of the original functions. This reduces the problem of deriving bounds on the adversarial risk to the problem of deriving risk bounds using standard learning-theoretic techniques. We then derive bounds on the Rademacher complexities of the transformed function classes, obtaining error rates on the same order as the generalization error of the original function classes. We also discuss extensions of our theory to multiclass classification and regression. Finally, we provide two algorithms for optimizing the adversarial risk bounds in the linear case, and discuss connections to regularization and distributional robustness.

Adversarial Risk Bounds via Function Transformation

TL;DR

This work presents a principled framework for bounding adversarial risk by transforming the loss-augmented predictor into transformed predictors whose standard generalization bounds apply. The supremum transform handles linear classifiers exactly, yielding explicit bounds that separate optimization error, data geometry, and perturbation strength, while the tree transform extends this idea to neural networks, enabling finite-sample risk bounds via Rademacher complexity. The authors extend the theory to multiclass classification and regression, and propose optimization schemes to minimize these adversarial-risk bounds in practical settings. The results illuminate how adversarial perturbations affect generalization and connect robustness to distributional-robustness concepts, with practical algorithms for robust linear and neural-network training. Overall, the paper provides a versatile toolkit for analyzing and improving adversarial robustness under standard learning-theoretic guarantees.

Abstract

We derive bounds for a notion of adversarial risk, designed to characterize the robustness of linear and neural network classifiers to adversarial perturbations. Specifically, we introduce a new class of function transformations with the property that the risk of the transformed functions upper-bounds the adversarial risk of the original functions. This reduces the problem of deriving bounds on the adversarial risk to the problem of deriving risk bounds using standard learning-theoretic techniques. We then derive bounds on the Rademacher complexities of the transformed function classes, obtaining error rates on the same order as the generalization error of the original function classes. We also discuss extensions of our theory to multiclass classification and regression. Finally, we provide two algorithms for optimizing the adversarial risk bounds in the linear case, and discuss connections to regularization and distributional robustness.

Paper Structure

This paper contains 42 sections, 39 theorems, 196 equations, 3 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem setup
  3. Risk and losses
  4. Function classes and Rademacher complexity
  5. Main results
  6. The supremum transformation and linear classification
  7. The tree transformation and neural networks
  8. A visualization of the tree transform
  9. Extension to multiclass classification
  10. Setup
  11. Multiclass results
  12. Linear classification
  13. Neural networks
  14. Extension to regression
  15. Setup
  16. ...and 27 more sections

Key Result

Proposition 1

Let $\ell(f, z)$ be a loss function that is monotonically decreasing in $yf(x)$. Then

Figures (3)

  • Figure 1: A visualization of $f(x + w)$. The input $x + w$ is fed up through the network.
  • Figure 2: A visualization of the function $g(x, y)$ of equation \ref{['eqnTreeStep1']}. Note that two different perturbations, $w^{(1)}$ and $w^{(2)}$, are fed upward through different paths in the network.
  • Figure 3: A visualization of the function $Tf(x, y)$ in equation \ref{['eqnTreeStep2']}. Note that four distinct perturbed inputs are fed through the network via different paths. The resulting tree-structured graph leads to the name "tree transform."

Theorems & Definitions (81)

  • Definition 1
  • Definition 2
  • Proposition 1
  • Remark 1
  • Proposition 2
  • Lemma 1
  • Corollary 1
  • Remark 2
  • Remark 3
  • Definition 3
  • ...and 71 more