Provable Unrestricted Adversarial Training without Compromise with Generalizability

TL;DR

This paper targets the dual challenges of defending against unrestricted adversarial examples (UAEs) and overcoming the traditional tradeoff between adversarial robustness and standard generalization. It introduces Provable Unrestricted Adversarial Training (PUAT), a framework that generates UAEs via an attacker and generator, and aligns the UAE distribution with the natural data distribution using a G-C-D GAN, while extending adversarial training with supervised loss and Mean Teacher consistency to ensure robust generalization. Theoretical results show that, with infinite data PUAT achieves $P_C = P_G = P$, eliminating the robustness-generalization conflict, and that with finite data, PUAT provides bounded generalization errors and a bounded UAE adversary gap that shrink as data increases. Empirically, PUAT consistently outperforms baselines across multiple datasets on both natural and adversarially perturbed inputs, improves the robustness-generalization tradeoff, leverages unlabeled data effectively, and demonstrates distributions alignment via visual analysis, albeit with longer training times than standard training but competitive against other GAN-based AT methods.

Abstract

Adversarial training (AT) is widely considered as the most promising strategy to defend against adversarial attacks and has drawn increasing interest from researchers. However, the existing AT methods still suffer from two challenges. First, they are unable to handle unrestricted adversarial examples (UAEs), which are built from scratch, as opposed to restricted adversarial examples (RAEs), which are created by adding perturbations bound by an $l_p$ norm to observed examples. Second, the existing AT methods often achieve adversarial robustness at the expense of standard generalizability (i.e., the accuracy on natural examples) because they make a tradeoff between them. To overcome these challenges, we propose a unique viewpoint that understands UAEs as imperceptibly perturbed unobserved examples. Also, we find that the tradeoff results from the separation of the distributions of adversarial examples and natural examples. Based on these ideas, we propose a novel AT approach called Provable Unrestricted Adversarial Training (PUAT), which can provide a target classifier with comprehensive adversarial robustness against both UAE and RAE, and simultaneously improve its standard generalizability. Particularly, PUAT utilizes partially labeled data to achieve effective UAE generation by accurately capturing the natural data distribution through a novel augmented triple-GAN. At the same time, PUAT extends the traditional AT by introducing the supervised loss of the target classifier into the adversarial loss and achieves the alignment between the UAE distribution, the natural data distribution, and the distribution learned by the classifier, with the collaboration of the augmented triple-GAN. Finally, the solid theoretical analysis and extensive experiments conducted on widely-used benchmarks demonstrate the superiority of PUAT.

Figures (7)

• Figure 1: Illustration of adversarial robustness against RAE and UAE. The blue dots and green triangles represent two classes of observed examples, respectively, and the gray ones represent the unobserved examples of the two classes. Each example is surrounded by a box representing its neighborhood with small distance. The purple stars represent RAEs while the red stars UAEs. The yellow curves are decision boundaries obtained by adversarial training.
• Figure 2: Architecture of PUAT. The attacker $A$ seeks the perturbation $z_a$, and the generator $G$ synthesizes the UAEs $\{ (\tilde{x}_g, y) \}$ based on the perturbation and the specified class $y$. In G-D GAN, the discriminator $D$ aims to adversarially distinguish the unobserved natural examples $\{ (x_g, y) \}$ generated by $G$ from the true natural examples $\{ (x, y)\}$ with loss $\mathcal{L}^{G,D}_{gan}$, while in C-D GAN, $D$ aims to adversarially distinguish the pseudo labeled examples $\{ (x, y_c)\}$ generated by the target classifier $C$ from $\{ (x, y)\}$ with loss $\mathcal{L}^{C,D}_{gan}$. The extended AT is conducted between $A$ and $C$ with the adversarial loss $\mathcal{L}^{C,A}_{adv}$ plus the supervised loss $\mathcal{L}^{C}_{nat}$.
• Figure 3: Pareto front curves of natural accuracy vs. robust accuracy on CIFAR10 under different adversarial attacks including (a) PGD-8/255, (b) AA, (c) GPGD-0.1, and (d) USong. Each curve consists of points generated by setting $\beta$ with various values.
• Figure 4: Effect of UAE and (un)labeled data. Subfigure (a) shows the natural accuracy and robust accuracy of PUAT under different $\lambda$ on CIFAR10. Subfigure (b) shows the natural accuracy and robust accuracy of PUAT and PUAT-w/o-unlabeled-data, and the Silhouette Coefficients, over different amount of labeled data on CIFAR10, where PUAT-w/o-unlabeled-data is a variant of PUAT without using unlabeled data during the training. The robust accuracy in both (a) and (b) is the average of the robust accuracies on PGD-8/255, AA, GPGD-0.1 and USong, which reflects the comprehensive adversarial robustness.
• Figure 5: Learning curves of PUAT with UAEs ($\lambda=10.0$) and without UAEs ($\lambda=0.0$). In both Subfigures (a) and (b), blue and green curves are the robust/natural accuracy curve of training with UAEs and its corresponding testing robust/natural accuracy curve, respectively, while orange and red curves are the robust/natural accuracy curve of training withouth UAEs and its corresponding testing robust/natural accuracy curve, respectively.

Theorems & Definitions (19)

• Definition 1: UAE
• Lemma 1
• Theorem 1
• proof
• Remark 1
• Theorem 2
• proof
• Remark 2
• Lemma 2
• Theorem 3