Variance-Based Pruning for Accelerating and Compressing Trained Networks

Abstract

Increasingly expensive training of ever larger models such as Vision Transfomers motivate reusing the vast library of already trained state-of-the-art networks. However, their latency, high computational costs and memory demands pose significant challenges for deployment, especially on resource-constrained hardware. While structured pruning methods can reduce these factors, they often require costly retraining, sometimes for up to hundreds of epochs, or even training from scratch to recover the lost accuracy resulting from the structural modifications. Maintaining the provided performance of trained models after structured pruning and thereby avoiding extensive retraining remains a challenge. To solve this, we introduce Variance-Based Pruning, a simple and structured one-shot pruning technique for efficiently compressing networks, with minimal finetuning. Our approach first gathers activation statistics, which are used to select neurons for pruning. Simultaneously the mean activations are integrated back into the model to preserve a high degree of performance. On ImageNet-1k recognition tasks, we demonstrate that directly after pruning DeiT-Base retains over 70% of its original performance and requires only 10 epochs of fine-tuning to regain 99% of the original accuracy while simultaneously reducing MACs by 35% and model size by 36%, thus speeding up the model by 1.44x. The code is available at: https://github.com/boschresearch/variance-based-pruning

Figures (8)

• Figure 1: Schematic visualization of the steps of Variance-Based Pruning. (a) Activation Statistics Computation, (b) Variance-Based Pruning, (c) Mean-Shift Compensation
• Figure 2: Accuracy retention before fine-tuning and final accuracy after 10 epochs of fine-tuning for different pruning rates applied to -Base.
• Figure 3: Activation variance for all neurons in the hidden layers throughout the network and marked cumulative variances.
• Figure 4: Learning curves over 100 epochs of fine-tuning after different structured pruning methods. VBP retains a performance lead over other methods throughout the entire training period.
• Figure 5: Accuracy retention across varying pruning ratios for different structured pruning methods applied to -Base. VBP consistently retains more accuracy across all pruning levels.