MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillation

TL;DR

The paper tackles robustness saturation and teacher-dependence in Adversarial Robustness Distillation by reframing the min-max ARD optimization. It introduces MMARD, which enriches the inner maximization with the teacher’s robust predictions to generate more informative adversarial examples and adds a triangular relationship in the outer minimization to better capture invariant mappings between natural and robust scenarios. Empirical results on CIFAR-10/100 with WideResNet teachers show state-of-the-art robust accuracy under AutoAttack and improved resilience to teacher choice, while maintaining competitive clean accuracy. The approach is plug-and-play and compatible with existing ARD methods, offering a practical route to stronger small-model robustness with less hyperparameter sensitivity.

Abstract

Adversarial Robustness Distillation (ARD) is a promising task to boost the robustness of small-capacity models with the guidance of the pre-trained robust teacher. The ARD can be summarized as a min-max optimization process, i.e., synthesizing adversarial examples (inner) & training the student (outer). Although competitive robustness performance, existing ARD methods still have issues. In the inner process, the synthetic training examples are far from the teacher's decision boundary leading to important robust information missing. In the outer process, the student model is decoupled from learning natural and robust scenarios, leading to the robustness saturation, i.e., student performance is highly susceptible to customized teacher selection. To tackle these issues, this paper proposes a general Min-Max optimization Adversarial Robustness Distillation (MMARD) method. For the inner process, we introduce the teacher's robust predictions, which drive the training examples closer to the teacher's decision boundary to explore more robust knowledge. For the outer process, we propose a structured information modeling method based on triangular relationships to measure the mutual information of the model in natural and robust scenarios and enhance the model's ability to understand multi-scenario mapping relationships. Experiments show our MMARD achieves state-of-the-art performance on multiple benchmarks. Besides, MMARD is plug-and-play and convenient to combine with existing methods.

Figures (4)

• Figure 1: (a) A toy illustration about the synthetic range of existing methods (left) and expected range (right). The red dot denotes the real label, the blue dot denotes the teacher's prediction for the natural example, and the orange dots denote synthetic training examples. (b) A toy experiment about the students' clean and robust performance of ARD goldblum2020adversarially and RSLAD zi2021revisiting with various teachers.
• Figure 2: (a) The pipeline of our outer minimization process to train the student model. In addition to the original individual example representation (purple box), we combine a triangular relationship to measure the models' mutual relations of the natural and robust scenarios (green box). (b) The existing individual data examples representation. The consistency regularization forces the student's prediction distribution close to the teacher's prediction distribution, which is susceptible to the influence of specific experimental settings. (c) Our triangular relationship of models' predictions on natural and corresponding adversarial examples. Our regularization transfers general invariant features to enhance the student's adaptability to different teachers.
• Figure 3: Diagrams of existing and our inner maximization process for synthesizing training examples. The training examples from existing methods are far from the teacher's decision boundary, containing less robust information. Our method reduces the discrepancy between the decision boundaries of the student and the teacher, thus promoting the robustness of the student.
• Figure 4: Students' performance about different combinations of inner maximization and outer minimization processes. RSLAD $\!\! + \!\!$ MMARD represents the combinations of RSLAD's inner process and MMARD's outer process. Eq (4) represents using only Equation \ref{['eq4']} as the outer process.