Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Better
MingWei Zhou, Xiaobing Pei
TL;DR
Adversarial training (AT) often trades natural accuracy for robustness, motivating a principled study of label design. The paper introduces a two-part framework—providing correct guidance and reducing learning difficulty—and develops CKTAT, which uses clean-model predictions with initialization and a temperature scaling to guide AT and reduce dense weight mixtures. Empirical results on CIFAR-10/100 show that CKTAT improves both natural and robust accuracy over strong baselines, with a tunable trade-off via a regularization parameter $\beta$. The work provides a unified lens on label-design strategies in AT and highlights how clean supervision can be leveraged to strengthen robustness without sacrificing generalization.$
Abstract
Adversarial training (AT) is an effective technique for enhancing adversarial robustness, but it usually comes at the cost of a decline in generalization ability. Recent studies have attempted to use clean training to assist adversarial training, yet there are contradictions among the conclusions. We comprehensively summarize the representative strategies and, with a focus on the multi - view hypothesis, provide a unified explanation for the contradictory phenomena among different studies. In addition, we conduct an in - depth analysis of the knowledge combinations transferred from clean - trained models to adversarially - trained models in previous studies, and find that they can be divided into two categories: reducing the learning difficulty and providing correct guidance. Based on this finding, we propose a new idea of leveraging clean training to further improve the performance of advanced AT methods.We reveal that the problem of generalization degradation faced by AT partly stems from the difficulty of adversarial training in learning certain sample features, and this problem can be alleviated by making full use of clean training.