Bag of Tricks for Adversarial Training

TL;DR

This paper tackles why adversarial training improvements often underperform by demonstrating that robustness is highly sensitive to seemingly minor training details. Through extensive ablations on CIFAR-10 under the $\ell_{\infty}$ threat model with $\epsilon=8/255$, the authors isolate the impact of tricks like weight decay, BN mode, label smoothing, and learning rate schedules, proposing a standardized baseline. They show that a baseline PGD-AT recipe and careful re-implementation of TRADES can surpass prior state-of-the-art AutoAttack results, highlighting that implementation confounds can masquerade as method gains. The work calls for rigorous, standardized benchmarking to ensure fair comparisons across AT methods and framework evaluations, with implications for reproducibility and progress in robustness research.

Abstract

Adversarial training (AT) is one of the most effective strategies for promoting model robustness. However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure. This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods. Surprisingly, we find that the basic settings (e.g., weight decay, training schedule, etc.) used in these methods are highly inconsistent. In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models. Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought. For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7%, which is probable to override the potential promotion induced by the proposed methods. We conclude a baseline training setting and re-implement previous defenses to achieve new state-of-the-art results. These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.

Figures (6)

• Figure 1: (a) Test accuracy w.r.t. different values of weight decay. The reported checkpoints correspond to the best PGD-10 accuracy rice2020overfitting. We test on two model architectures, and highlight (with red circles) three most commonly used weight decays in previous work; (b) Curves of test accuracy w.r.t. training epochs, where the model is WRN-34-10. We set weight decay be $1\times 10^{-4}$, $2\times 10^{-4}$, and $5\times 10^{-4}$, respectively. We can observe that smaller weight decay can learn faster but also more tend to overfit w.r.t. the robust accuracy. In Fig. \ref{['fig:wd_appendix']}, we early decay the learning rate before the models overfitting, but weight decay of $5\times 10^{-4}$ still achieve better robustness.
• Figure 2: Clean accuracy vs. PGD-10 accuracy for different model architectures. The circle sizes are proportional to the number of parameters that specified in Table \ref{['tableappendix3']}.
• Figure 3: Test accuracy (%) under different batch size and learning rate (l.r.) on CIFAR-10. The basic l.r. is $0.1$, while the scaled l.r. is, e.g., $0.2$ for batch size $256$, and $0.05$ for batch size $64$.
• Figure 3: Random normal cross-sections of the decision boundary for PGD-AT with different weight decay. The model architecture is WRN-34-10. Following the examples in moosavi2019robustness, we craft PGD-10 perturbation as the normal direction $v$, and $r$ be a random direction, under the $\ell_{\infty}$ constraint of $8/255$. The values of x-axis and y-axis represent the multiplied scale factors.
• Figure 4: Curves of test accuracy w.r.t. training epochs, where the model is WRN-34-10. Here we early decay the learning rate at 40 and 45 epochs for the cases of weight decay $1\times 10^{-4}$ and $2\times 10^{-4}$, just before they overfitting. We can see that the models can achieve the same clean accuracy as weight decay $5\times 10^{-4}$, but still worse robustness.