S2AP: Score-space Sharpness Minimization for Adversarial Pruning

TL;DR

S2AP introduces score-space sharpness minimization for adversarial pruning, reformulating mask search in the importance-score space and perturbing scores to reduce local sharpness in the robust loss. The method is a plug-in that perturbs scores with a bound $\gamma$ per layer and optimizes a max-min objective, followed by finetuning with weight-space sharpness minimization. Across multiple architectures, datasets, and sparsity levels, S2AP consistently improves adversarial robustness with limited clean-accuracy loss, while reducing score-space sharpness and enhancing mask stability. These findings suggest that stabilizing the score-space landscape yields more reliable pruning decisions and robust sparse models, albeit with a modest computational overhead and hyperparameter considerations for $\gamma$.

Abstract

Adversarial pruning methods have emerged as a powerful tool for compressing neural networks while preserving robustness against adversarial attacks. These methods typically follow a three-step pipeline: (i) pretrain a robust model, (ii) select a binary mask for weight pruning, and (iii) finetune the pruned model. To select the binary mask, these methods minimize a robust loss by assigning an importance score to each weight, and then keep the weights with the highest scores. However, this score-space optimization can lead to sharp local minima in the robust loss landscape and, in turn, to an unstable mask selection, reducing the robustness of adversarial pruning methods. To overcome this issue, we propose a novel plug-in method for adversarial pruning, termed Score-space Sharpness-aware Adversarial Pruning (S2AP). Through our method, we introduce the concept of score-space sharpness minimization, which operates during the mask search by perturbing importance scores and minimizing the corresponding robust loss. Extensive experiments across various datasets, models, and sparsity levels demonstrate that S2AP effectively minimizes sharpness in score space, stabilizing the mask selection, and ultimately improving the robustness of adversarial pruning methods.

Figures (17)

• Figure 1: Effects of S2AP on a ResNet18 CIFAR10 model. (a) Improved robust accuracy of pruned models. (b) Enhanced mask stability (quantified as Hamming distance $h$, i.e., measuring how much each mask $\boldsymbol{m}_t$ across pruning epochs changes compared to the first computed mask $\boldsymbol{m}_0$). We subtract and plot $h_{orig}-h_{S2AP\xspace}$, thus yielding positive values where S2AP is more stable (green area), and negative values vice versa (red area). S2AP enhances mask stability, particularly after pruning epoch 5 when warm-up ends and explicit sharpness minimization begins. (c) Minimized sharpness in the robust loss landscape (where the largest eigenvalue $\lambda_{max}$ indicates more sharpness).
• Figure 2: Score-space sharpness measured via largest eigenvalue $\lambda_{max}$ over pruning epochs for HARP on WideResNet28-4 and CIFAR10.
• Figure 3: Score-space sharpness measured as difference of perturbed and reference loss values on ResNet18 and WideResNet28-4 CIFAR10 pruned models.
• Figure 4: The Hamming distance for WideResNet28-4 on CIFAR10. In (a) the single hamming distance from epoch $5$ of S2AP and Orig. HARP. Lower curves indicate higher stability. In (b), the results from the four (a) subplots by subtracting each Original-S2AP curve, thus yielding a positive-green (negative-red) area where S2AP (Original) methods are more stable.
• Figure 5: Robustness of S2AP pruning masks found using different $\gamma$ values bounding the score perturbation.