Improving Fast Adversarial Training via Self-Knowledge Guidance

TL;DR

This work tackles the imbalance and misalignment phenomena in fast adversarial training (FAT) by introducing Self-Knowledge Guided FAT (SKG-FAT). It combines self-knowledge guided regularization (CWR and AGR) with self-knowledge guided label relaxation (SKLR) to selectively emphasize training on classes and groups with training-state signals, without adding hyperparameters. The proposed approach yields improved adversarial robustness across CIFAR-10/100, Tiny ImageNet, and ImageNet-100 while preserving clean accuracy and maintaining efficiency, often outperforming state-of-the-art FAT methods and avoiding catastrophic overfitting. The study provides practical insights into the role of per-class and per-group guidance in FAT, offering scalable techniques that can adapt to large-scale datasets and diverse architectures.

Abstract

Adversarial training has achieved remarkable advancements in defending against adversarial attacks. Among them, fast adversarial training (FAT) is gaining attention for its ability to achieve competitive robustness with fewer computing resources. Existing FAT methods typically employ a uniform strategy that optimizes all training data equally without considering the influence of different examples, which leads to an imbalanced optimization. However, this imbalance remains unexplored in the field of FAT. In this paper, we conduct a comprehensive study of the imbalance issue in FAT and observe an obvious class disparity regarding their performances. This disparity could be embodied from a perspective of alignment between clean and robust accuracy. Based on the analysis, we mainly attribute the observed misalignment and disparity to the imbalanced optimization in FAT, which motivates us to optimize different training data adaptively to enhance robustness. Specifically, we take disparity and misalignment into consideration. First, we introduce self-knowledge guided regularization, which assigns differentiated regularization weights to each class based on its training state, alleviating class disparity. Additionally, we propose self-knowledge guided label relaxation, which adjusts label relaxation according to the training accuracy, alleviating the misalignment and improving robustness. By combining these methods, we formulate the Self-Knowledge Guided FAT (SKG-FAT), leveraging naturally generated knowledge during training to enhance the adversarial robustness without compromising training efficiency. Extensive experiments on four standard datasets demonstrate that the SKG-FAT improves the robustness and preserves competitive clean accuracy, outperforming the state-of-the-art methods.

Figures (12)

• Figure 1: Class-wise training results on CIFAR-10 training set. (a) Results obtained by FGSM-RS. (b) Results obtained by PGI-BP.
• Figure 2: Schematic diagram of the perspective of accuracy alignment taxonomy.
• Figure 3: Training results under the perspective of accuracy alignment using FGSM-RS on CIFAR-100 and ImageNet-100. "Avg" denotes average clean or robust accuracy. The bottom row shows the number of classes in different groups that change with training. (a) Results on CIFAR-100. (b) Results on ImageNet-100.
• Figure 4: Transition of classes with GCBR or BCGR on CIFAR100 dataset during FGSM-RS training.
• Figure 5: The number of classes included in each group of the perspective of accuracy alignment during FGSM-RS training. (a)-(c) CIFAR-100 results with different models. (d) Results on ImageNet-1K with ResNet-50.