Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quadratic Upper Bound for Boosting Robustness

Euijin You, Hyang-Won Lee

TL;DR

This work tackles the robustness-efficiency tradeoff in adversarial training by deriving a quadratic upper bound ($QUB$) on the AT loss, leveraging the convexity of cross-entropy in logits. The $QUB$ loss can be integrated into existing FAT pipelines (yielding either $QUB$-static or $QUB$-decreasing variants) to improve robustness with modest training-time overhead. Through extensive experiments on CIFAR-10/100 and Tiny ImageNet, the authors show that $QUB$-augmented FAT generally enhances robust accuracy, while the $QUB$-decreasing schedule better preserves standard accuracy. Analyses of loss landscapes, Hessian eigenvalues, and adversarial sparsity support the interpretation that $QUB$ smooths the loss landscape and expands the robust region, contributing to improved resilience against unseen attacks.

Abstract

Fast adversarial training (FAT) aims to enhance the robustness of models against adversarial attacks with reduced training time, however, FAT often suffers from compromised robustness due to insufficient exploration of adversarial space. In this paper, we develop a loss function to mitigate the problem of degraded robustness under FAT. Specifically, we derive a quadratic upper bound (QUB) on the adversarial training (AT) loss function and propose to utilize the bound with existing FAT methods. Our experimental results show that applying QUB loss to the existing methods yields significant improvement of robustness. Furthermore, using various metrics, we demonstrate that this improvement is likely to result from the smoothened loss landscape of the resulting model.

Quadratic Upper Bound for Boosting Robustness

TL;DR

This work tackles the robustness-efficiency tradeoff in adversarial training by deriving a quadratic upper bound () on the AT loss, leveraging the convexity of cross-entropy in logits. The loss can be integrated into existing FAT pipelines (yielding either -static or -decreasing variants) to improve robustness with modest training-time overhead. Through extensive experiments on CIFAR-10/100 and Tiny ImageNet, the authors show that -augmented FAT generally enhances robust accuracy, while the -decreasing schedule better preserves standard accuracy. Analyses of loss landscapes, Hessian eigenvalues, and adversarial sparsity support the interpretation that smooths the loss landscape and expands the robust region, contributing to improved resilience against unseen attacks.

Abstract

Fast adversarial training (FAT) aims to enhance the robustness of models against adversarial attacks with reduced training time, however, FAT often suffers from compromised robustness due to insufficient exploration of adversarial space. In this paper, we develop a loss function to mitigate the problem of degraded robustness under FAT. Specifically, we derive a quadratic upper bound (QUB) on the adversarial training (AT) loss function and propose to utilize the bound with existing FAT methods. Our experimental results show that applying QUB loss to the existing methods yields significant improvement of robustness. Furthermore, using various metrics, we demonstrate that this improvement is likely to result from the smoothened loss landscape of the resulting model.
Paper Structure (25 sections, 4 theorems, 38 equations, 5 figures, 6 tables, 2 algorithms)

This paper contains 25 sections, 4 theorems, 38 equations, 5 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial Attack
  4. Adversarial Training
  5. Fast Adversarial Training
  6. Quadratic Upper Bound for AT
  7. Deriving the Bound
  8. Interpretation of QUB Loss
  9. Training Strategy
  10. Experiments
  11. Experimental Settings
  12. Impact of QUB on Accuracy
  13. Flatness of the Loss Landscape
  14. Adversarial Sparsity
  15. Conclusion
  16. ...and 10 more sections

Key Result

Lemma 1

The AT loss function is upper-bounded as follows: where $\nabla_f \mathcal{L}$ is the gradient of the loss with respect to the logit $f$ and $\|\bm{H}\|_2$ is the $L_2$ norm of the Hessian matrix of the loss with respect to the logit, evaluated at some point between $f(x)$ and $f(x+\delta)$.

Figures (5)

  • Figure 1: Loss landscape for a specific sample: (a) model trained with FGSM-CKPT and (b) with FGSM-CKPT + QUB. The left side shows colors based on the loss value, and the right side shows colors based on prediction accuracy.
  • Figure 2: Average dominant eigenvalue for each method. Models trained with QUB loss show smaller dominant eigenvalues.
  • Figure 3: Comparing FGSM-CKPT (blue line) and FGSM-CKPT + QUB (orange line) for each attack budget shows that the sparsity value with QUB is consistently higher in all ranges.
  • Figure 4: Sparsity values with and without QUB for each method. Using QUB consistently results in higher values across all methods.
  • Figure 5: Accuracy improvement from applying QUB to adversarial training using (a) PGD-AT and (b) N-FGSM. While QUB-decreasing shows smaller AA gains compared to QUB-static, it better preserves standard accuracy. As the perturbation strength $\epsilon$ increases, the improvement gradually diminishes, reflecting the characteristics of QUB.

Theorems & Definitions (4)

  • Lemma 1
  • Lemma 2
  • Lemma 3: Taylor's Theorem, taylors_theorem
  • Lemma 4: linear_algebra