Boosting Pruned Networks with Linear Over-parameterization

TL;DR

This work tackles accuracy loss after structured pruning by introducing linear over-parameterization of compact layers during fine-tuning, followed by re-parameterization to the slim model. It augments this with similarity-preserving knowledge distillation to preserve feature learning, guiding the over-parameterized blocks to maintain data-to-data similarities. Empirical results on CIFAR-10 and ImageNet demonstrate substantial improvements over vanilla fine-tuning, especially at high pruning ratios, including notable gains under extreme pruning. The approach is compatible with multiple pruning strategies and can be integrated into pruning pipelines to enhance overall effectiveness.

Abstract

Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. However, too few remaining parameters in pruned networks inevitably bring a great challenge to fine-tuning to restore accuracy. To address this challenge, we propose a novel method that first linearly over-parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters and then re-parameterizes them to the original layers after fine-tuning. Specifically, we equivalently expand the convolution/linear layer with several consecutive convolution/linear layers that do not alter the current output feature maps. Furthermore, we utilize similarity-preserving knowledge distillation that encourages the over-parameterized block to learn the immediate data-to-data similarities of the corresponding dense layer to maintain its feature learning ability. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet which significantly outperforms the vanilla fine-tuning strategy, especially for large pruning ratio.

Figures (4)

• Figure 1: An overall workflow of our method. (a) A pretrained network. (b) A pruned network. (c) Each pruned layer is linearly over-parameterized by matrix decomposition. (d) The fine-tuned over-parameterized model is contracted back to the slim model. To preserve the feature extraction ability of each layer after over-parameterization, pairwise activation similarity within each input mini-batch is used to supervise the training of the over-parameterized network.
• Figure 2: An illustration of the over-parameterization process of a convolutional layer.
• Figure 3: Top-1 accuracy(%) on the test set of CIFAR-10 based on ResNet-56.
• Figure 4: Top-1 accuracy(%) on the test set of CIFAR-10 based on MobileNetV2.