Revisiting Semi-supervised Adversarial Robustness via Noise-aware Online Robust Distillation

TL;DR

This work presents SNORD - a simple yet effective framework that introduces contemporary semi-supervised learning techniques into the realm of adversarial training, and showcases impressive, state-of-the-art performance across diverse datasets and labeling budgets, all without the need for pretrained models.

Abstract

The robust self-training (RST) framework has emerged as a prominent approach for semi-supervised adversarial training. To explore the possibility of tackling more complicated tasks with even lower labeling budgets, unlike prior approaches that rely on robust pretrained models, we present SNORD - a simple yet effective framework that introduces contemporary semi-supervised learning techniques into the realm of adversarial training. By enhancing pseudo labels and managing noisy training data more effectively, SNORD showcases impressive, state-of-the-art performance across diverse datasets and labeling budgets, all without the need for pretrained models. Compared to full adversarial supervision, SNORD achieves a 90% relative robust accuracy under epsilon = 8/255 AutoAttack, requiring less than 0.1%, 2%, and 10% labels for CIFAR-10, CIFAR-100, and TinyImageNet-200, respectively. Additional experiments confirm the efficacy of each component and demonstrate the adaptability of integrating SNORD with existing adversarial pretraining strategies to further bolster robustness.

Figures (4)

• Figure 1: Performance comparison of SSL adversarial training techniques. The RST pipeline is widely adopted by current SSL adversarial training methods. While ACL and DynACL++ incorporate robust pretrained models to enhance the basic RST, their achievements remain suboptimal due to the intrinsic limitations of the RST (further elaborated in \ref{['fig:fig15']}). After addressing these issues, our SNORD framework outperforms all these methods by a large margin across diverse labeling budgets and datasets. Notably, on the CIFAR-10 dataset, SNORD attains comparable results to fully adversarial training methods like TRADES and TRADES-AWP but requires only 0.2% of the labeling effort.
• Figure 2: Revisiting SSL Adversarial Training. (a) Addressing the limitations of the two-stage RST method involves enhancing initial pseudo label (PL) quality and efficiently managing noisy data in downstream adversarial training. (b) On CIFAR-100, the state-of-the-art RST-based method (DynACL++) still generates lower-quality PLs initially, leading to suboptimal Standard Accuracy (SA) and Adversarial Accuracy (AA) after conventional adversarial training, compared to a fully adversarially trained oracle model. Our approach, utilizing an advanced SSL algorithm for PL generation, significantly improves performance under identical downstream training conditions. (c) We analyze the impact of different downstream adversarial training strategies on CIFAR-100 with equivalent noisy PLs. The y-axis indicates the relative performance compared to the oracle case—the performance under adversarial training using fully labeled data. The results show that the basic RST method with hard labels underperforms due to inaccurate PLs, resulting in the worst AA. Strategies combining adversarial pretraining with soft distillation maintain higher AA but at the cost of reduced SA in a low labeling regime. Conversely, our proposed noise-aware rectification and online robust distillation effectively overcome these issues, achieving superior SA and AA. Further details about our method are provided in \ref{['sec:method']}.
• Figure 3: Semi-supervised Noise-aware Online Robust Distillation (SNORD). Our SNORD framework aims to address two important yet previously overlooked issues associated with noisy pseudo labels in the field of semi-supervised adversarial training. Firstly, it employs advanced SSL algorithms to improve the quality of pseudo labels (\ref{['ssec:ssl_algo']}). Secondly, we introduce Noise-aware Rectification (\ref{['ssec:nar']}) and Online Robust Distillation (\ref{['ssec:ord']}) to enhance the learning capabilities of the downstream adversarial robust model in the context of handling noisy estimated pseudo labels.
• Figure 4: Sensitivity analyses of our NAR methods. The result underscores the importance of using sampling rather than argmax to obtain pseudo labels and the robustness of our approach over a wide range of hyper-parameters.