On Using Certified Training towards Empirical Robustness

TL;DR

This work interrogates whether certified training, traditionally aimed at verifiable robustness, can bolster empirical robustness against adversarial attacks. By systematically studying expressive losses (notably Exp-IBP) and the lightweight ForwAbs regularizer, the authors demonstrate that, when tuned for empirical goals, certified-training techniques can mitigate catastrophic overfitting for single-step attacks and sometimes rival or exceed multi-step baselines under certain conditions. The results highlight a nuanced trade-off between empirical performance and verifiability, showing promise on easier datasets and shallower networks but limited gains on harder tasks like CIFAR-100. The study also introduces a scalable regularizer that reduces the overhead of IBP-based methods, suggesting practical avenues to integrate certified principles into empirical defenses. Collectively, the work advances understanding of how certified training can contribute to empirical robustness and identifies avenues for future improvements in certified-training algorithms.

Abstract

Adversarial training is arguably the most popular way to provide empirical robustness against specific adversarial examples. While variants based on multi-step attacks incur significant computational overhead, single-step variants are vulnerable to a failure mode known as catastrophic overfitting, which hinders their practical utility for large perturbations. A parallel line of work, certified training, has focused on producing networks amenable to formal guarantees of robustness against any possible attack. However, the wide gap between the best-performing empirical and certified defenses has severely limited the applicability of the latter. Inspired by recent developments in certified training, which rely on a combination of adversarial attacks with network over-approximations, and by the connections between local linearity and catastrophic overfitting, we present experimental evidence on the practical utility and limitations of using certified training towards empirical robustness. We show that, when tuned for the purpose, a recent certified training algorithm can prevent catastrophic overfitting on single-step attacks, and that it can bridge the gap to multi-step baselines under appropriate experimental settings. Finally, we present a conceptually simple regularizer for network over-approximations that can achieve similar effects while markedly reducing runtime.

Figures (24)

• Figure 1: IBP loss of adversarial training schemes on CIFAR-10, setup from table \ref{['fig-table-long-schedule']}.
• Figure 2: IBP certified robustness attained by MTL-IBP and Exp-IBP on the PreActResNet18 training setup from Jorge2022. Validation results on CIFAR-10 under perturbations of $\epsilon=8/255$.
• Figure 3: Sensitivity of the expressive losses on a toy network of varying depth, with $\alpha_{\text{Exp-IBP}}=10^{-1}$.
• Figure 4: IBP loss over epochs (left), box plots (10 runs) for the training time of an epoch (right), setup as figure \ref{['fig-ibp-accuracy-prn18']}.
• Figure 5: The use of certified training techniques on top of FGSM can prevent CO for PreActResNet18 on the CIFAR-10 test set under perturbations of $\epsilon=8/255$ and $\epsilon=24/255$.

Theorems & Definitions (3)

• Definition 3.1
• Definition 3.2
• Definition A.1