Adaptive Pruning with Module Robustness Sensitivity: Balancing Compression and Robustness

TL;DR

This work tackles the trade-off between neural network compression and adversarial robustness by introducing Module Robust Pruning and Fine-Tuning (MRPF), a framework guided by the Module Robustness Sensitivity (MRS) metric. MRPF adaptively allocates pruning across layers based on robustness impact and couples pruning with adversarial fine-tuning (notably TRADES) to preserve robust decision boundaries. Across CIFAR-10/100, SVHN, and Tiny-ImageNet on ResNet, VGG, and MobileViT, MRPF demonstrates improved adversarial robustness and competitive accuracy at various sparsity levels, outperforming state-of-the-art structured pruning methods in balancing robustness, accuracy, and compression. The framework is practical, scalable, and generalizable to diverse architectures, offering a principled path to robust, efficient pruned networks and motivating extensions to larger models and transformer-based backbones.

Abstract

Neural network pruning has traditionally focused on weight-based criteria to achieve model compression, frequently overlooking the crucial balance between adversarial robustness and accuracy. Existing approaches often fail to preserve robustness in pruned networks, leaving them more susceptible to adversarial attacks. This paper introduces Module Robustness Sensitivity (MRS), a novel metric that quantifies layer-wise sensitivity to adversarial perturbations and dynamically informs pruning decisions. Leveraging MRS, we propose Module Robust Pruning and Fine-Tuning (MRPF), an adaptive pruning algorithm compatible with any adversarial training method, offering both flexibility and scalability. Extensive experiments on SVHN, CIFAR, and Tiny-ImageNet across diverse architectures, including ResNet, VGG, and MobileViT, demonstrate that MRPF significantly enhances adversarial robustness while maintaining competitive accuracy and computational efficiency. Furthermore, MRPF consistently outperforms state-of-the-art structured pruning methods in balancing robustness, accuracy, and compression. This work establishes a practical and generalizable framework for robust pruning, addressing the long-standing trade-off between model compression and robustness preservation.

Figures (5)

• Figure 1: The impact of different pruning criteria (random, magnitude, Hessian, Taylor, LAMP) on accuracy and robustness for ResNet-50 on CIFAR-10. Both accuracy and robustness decline as the model becomes sparser, especially at high compression ratios. The choice of pruning criterion has limited influence on the final performance.
• Figure 2: Illustration of the proposed Module Robust Pruning and Fine-Tuning (MRPF) method. The process starts with a pre-trained network, generates adversarial examples, calculates layer-wise Module Robustness Sensitivity (MRS), and allocates pruning ratios dynamically. Critical layers are retained during pruning(layers with lower MRS values are pruned more aggressively), and adversarial fine-tuning ensures the pruned network maintains both robustness and accuracy.
• Figure 3: MRS (Module Robustness Sensitivity) scores distribution across layers of ResNet-18 evaluated on CIFAR-10, CIFAR-100, and SVHN datasets. The y-axis uses a logarithmic scale to emphasize the variations in MRS values across layers, revealing the consistent trend of robustness-critical layers across datasets. Deeper layers generally exhibit higher sensitivity, underscoring their significant role in maintaining adversarial robustness, while early and intermediate layers display lower but crucial sensitivity to adversarial perturbations.
• Figure 4: Impact of various pruning criteria on accuracy and robustness across different architectures and datasets. The figures illustrate the performance under different pruning methods (random, magnitude, Hessian, Taylor, LAMP) for ResNet-50, MobileViT-xs, and VGG-16 on CIFAR-10 and CIFAR-100. Regardless of the pruning method, similar accuracy and robustness are achieved after fine-tuning.
• Figure 5: Impact of different optimizers (SGD and ADAM) on pruned models for VGG-16-CIFAR-100 under different pruning strategies. The left figure shows random pruning, and the right figure shows magnitude pruning.