Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective

TL;DR

This work tackles catastrophic overfitting in fast adversarial training (FAT) by introducing an example taxonomy that reveals optimization imbalance between inner and outer FAT steps. It develops ETA, a comprehensive FAT enhancement consisting of batch momentum initialization, dynamic label relaxation, taxonomy driven loss, and catastrophical overfitting aware loss adaptation (COLA), yielding improved robustness across CIFAR-10/100, Tiny ImageNet, and ImageNet-100 with competitive clean accuracy. The authors provide extensive ablations, hyperparameter analyses, and visualization studies to support the effectiveness and stability of ETA, and show COLA can be plugged into other FAT methods for additional gains. Overall, ETA offers a principled path to stabilize FAT, concentrate training losses, and achieve state-of-the-art robustness with practical efficiency gains.

Abstract

While adversarial training is an effective defense method against adversarial attacks, it notably increases the training cost. To this end, fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. However, FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. However, the cause of catastrophic overfitting remains unclear and lacks exploration. In this paper, we present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT. Furthermore, we investigated the impact of varying degrees of training loss, revealing a correlation between training loss and catastrophic overfitting. Based on these observations, we redesign the loss function in FAT with the proposed dynamic label relaxation to concentrate the loss range and reduce the impact of misclassified examples. Meanwhile, we introduce batch momentum initialization to enhance the diversity to prevent catastrophic overfitting in an efficient manner. Furthermore, we also propose Catastrophic Overfitting aware Loss Adaptation (COLA), which employs a separate training strategy for examples based on their loss degree. Our proposed method, named example taxonomy aware FAT (ETA), establishes an improved paradigm for FAT. Experiment results demonstrate our ETA achieves state-of-the-art performance. Comprehensive experiments on four standard datasets demonstrate the competitiveness of our proposed method.

Figures (13)

• Figure 1: Catastrophic overfitting and label flipping. We adopt the ResNet18 to perform FGSM-AT FGSM and FGSM-RS FGSMRS on the CIFAR-10 dataset with perturbation budget $\epsilon=8/255$. The train and test robust accuracy is evaluated by the FGSM with $\alpha=8/255$ and PGD-10 with $\alpha=2/255$. The red line denotes the proportion of Case 4 (as Fig. \ref{['NumCO']}) to all misclassified examples, which infers the proportion of label flipping examples explodes once catastrophic overfitting occurs.
• Figure 2: The proposed taxonomy of training examples in FAT. The five cases in this figure correspond to Case 1 to Case 5, from top to bottom.
• Figure 3: Numbers of the five cases during training. The sum of the five cases for each epoch equals the number of examples in the training dataset.
• Figure 4: Influences of different case examples on training stability and accuracy. (Left) Clean accuracy. (Right) Robust accuracy.
• Figure 5: The change in the number of examples in different loss intervals when performing FGSM-RS FGSMRS.