Holistic Adversarially Robust Pruning

TL;DR

The paper tackles the challenge of preserving adversarial robustness while aggressively pruning neural networks. It introduces HARP, a holistic pruning framework that learns per-layer compression rates and connection scores during fine-tuning using a dynamic regularization schedule. Through a joint optimization framework with differentiable pruning and STE, HARP achieves substantial parameter reduction while maintaining natural accuracy and robustness, outperforming prior robust pruning methods at high sparsity. The work highlights the importance of non-uniform, layer-aware compression for robust deployment on resource-constrained hardware.

Abstract

Neural networks can be drastically shrunk in size by removing redundant parameters. While crucial for the deployment on resource-constraint hardware, oftentimes, compression comes with a severe drop in accuracy and lack of adversarial robustness. Despite recent advances, counteracting both aspects has only succeeded for moderate compression rates so far. We propose a novel method, HARP, that copes with aggressive pruning significantly better than prior work. For this, we consider the network holistically. We learn a global compression strategy that optimizes how many parameters (compression rate) and which parameters (scoring connections) to prune specific to each layer individually. Our method fine-tunes an existing model with dynamic regularization, that follows a step-wise incremental function balancing the different objectives. It starts by favoring robustness before shifting focus on reaching the target compression rate and only then handles the objectives equally. The learned compression strategies allow us to maintain the pre-trained model natural accuracy and its adversarial robustness for a reduction by 99% of the network original size. Moreover, we observe a crucial influence of non-uniform compression across layers.

Figures (13)

• Figure 1: Overview of pruning weights of a model for [10] (left) and (right) with -10 adversarial training. Solid lines show the natural accuracy of , , and (\ref{['tab:abbr-summary']}). Dashed lines represent the robustness against .
• Figure 2: 's step-wise regularization of pruning objectives for on [10] with a target sparsity of $\perc{99}$.
• Figure 3: Comparing -10 adversarial robustness of , , , and on models learned on [10]. All pre-trained models are provided by Lee2022MAD.
• Figure 4: Strategy comparison for pruning a with target 99.9 sparsity on [10].
• Figure 5: Different $\gamma$ step-size for pruning 99.9 weights of a by on [10].