Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge

TL;DR

The Dynamic Guidance Adversarial Distillation framework directly tackles the challenge of differential sample importance, with a keen focus on rectifying the teacher model's misclassifications, establishing it as a promising approach for enhancing both the robustness and accuracy of student models in adversarial settings.

Abstract

In the realm of Adversarial Distillation (AD), strategic and precise knowledge transfer from an adversarially robust teacher model to a less robust student model is paramount. Our Dynamic Guidance Adversarial Distillation (DGAD) framework directly tackles the challenge of differential sample importance, with a keen focus on rectifying the teacher model's misclassifications. DGAD employs Misclassification-Aware Partitioning (MAP) to dynamically tailor the distillation focus, optimizing the learning process by steering towards the most reliable teacher predictions. Additionally, our Error-corrective Label Swapping (ELS) corrects misclassifications of the teacher on both clean and adversarially perturbed inputs, refining the quality of knowledge transfer. Further, Predictive Consistency Regularization (PCR) guarantees consistent performance of the student model across both clean and adversarial inputs, significantly enhancing its overall robustness. By integrating these methodologies, DGAD significantly improves upon the accuracy of clean data and fortifies the model's defenses against sophisticated adversarial threats. Our experimental validation on CIFAR10, CIFAR100, and Tiny ImageNet datasets, employing various model architectures, demonstrates the efficacy of DGAD, establishing it as a promising approach for enhancing both the robustness and accuracy of student models in adversarial settings.

Figures (2)

• Figure 1: The overview of Dynamic Guidance Adversarial Distillation (DGAD) framework. The DGAD framework refines adversarial distillation by employing a strategic approach: Misclassification-Aware Partitioning (MAP) categorizes inputs for tailored learning, Error-corrective Label Swapping (ELS) fixes teacher’s mispredictions, and Predictive Consistency Regularization (PCR) maintains learning uniformity. Together, these methods improve student model accuracy and robustness. $S(\cdot)$ and $T(\cdot)$ are the predictions of the student and teacher models, while $\hat{T}(\cdot)$ is the corrected teacher predictions after ELS.
• Figure 2: Necessity of dynamically varying AD loss weights for individual samples. We compare the performance of AdaAD huang2023boosting, which originally proposes to employ a static weight $\alpha$ in (\ref{['eq:adaad']}), against our Dynamic Guidance Adversarial Distillation (DGAD) that adapts the weight dynamically per sample as in \ref{['eq:map']}. To validate the importance of dynamical weights, we adjust $\alpha$ for AdaAD and compare it across clean and adversarial scenarios. The blue solid line represents AdaAD's performance with a fixed $\alpha$ across all samples, while the red dotted line indicates DGAD's performance, showing improved accuracy due to its dynamic weighting approach.