RegMix: Adversarial Mutual and Generalization Regularization for Enhancing DNN Robustness

TL;DR

RegMix addresses adversarial robustness by introducing Adversarial Mutual Regularization (AMR) and Adversarial Generalization Regularization (AGR) to adversarial training.AMR employs a pair of weighted KL-divergence terms between final and initial adversarial outputs to allocate primary and auxiliary roles, while AGR adds a clean-output target to improve generalization under attack.Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet show that RegMix improves robustness against PGD and AutoAttack, with faster convergence and consistent gains across architectures and perturbation budgets.The approach yields smoother loss landscapes and probability distributions closer to ground truth, suggesting practical impact for deploying robust DNNs in security-sensitive settings.Overall, RegMix provides a principled, distillation-inspired framework that enhances defense strength against both similar-scale and stronger adversarial perturbations.

Abstract

Adversarial training is the most effective defense against adversarial attacks. The effectiveness of the adversarial attacks has been on the design of its loss function and regularization term. The most widely used loss function in adversarial training is cross-entropy and mean squared error (MSE) as its regularization objective. However, MSE enforces overly uniform optimization between two output distributions during training, which limits its robustness in adversarial training scenarios. To address this issue, we revisit the idea of mutual learning (originally designed for knowledge distillation) and propose two novel regularization strategies tailored for adversarial training: (i) weighted adversarial mutual regularization and (ii) adversarial generalization regularization. In the former, we formulate a decomposed adversarial mutual Kullback-Leibler divergence (KL-divergence) loss, which allows flexible control over the optimization process by assigning unequal weights to the main and auxiliary objectives. In the latter, we introduce an additional clean target distribution into the adversarial training objective, improving generalization and enhancing model robustness. Extensive experiments demonstrate that our proposed methods significantly improve adversarial robustness compared to existing regularization-based approaches.

Figures (9)

• Figure 1: An overview of the proposed RegMix framework. AE denotes the adversarial example, BP stands for backpropagation, and CLN represents the clean output. $I_{\text{adv}}$ and $F_{\text{adv}}$ indicate the initial (randomly initialized perturbation without gradient) and final (gradient-based) adversarial outputs, respectively. $y$ is the ground-truth label. The two core components of our method, Adversarial Mutual Regularization (AMR) and Adversarial Generalization Regularization (AGR), are introduced in Sections \ref{['Adversarial mutual']} and \ref{['Adversarial Generalization']}, respectively. AT refers to adversarial training.
• Figure 2: Robust accuracy under PGD-10 attack on CIFAR-10 and CIFAR-100. FGSM-AGR (with the generalization term) consistently outperforms FGSM-SAR (without the generalization term) across both datasets.
• Figure 3: Comparison between our methods (FGSM-AMR / FGSM-AGR) and the SOTA baseline (FGSM-PGK) on CIFAR-10 under PGD-10 attack. This plot shows both robustness (PGD-10 attack) and training cost (epochs and time in minutes). Our experiments were conducted on a single NVIDIA RTX 3090 Ti GPU.
• Figure 4: Robustness evaluation on CIFAR-10 under varying $\ell_\infty$ perturbation bounds using ResNet-18. (a) Results under PGD-10 attacks. (b) Robust accuracy under AutoAttack for different PGD-2-AT based defenses. Our methods show consistent superiority robustness under stronger perturbations.
• Figure 5: FGSM-PGI