Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhanced Sparsification via Stimulative Training

Shengji Tang, Weihao Lin, Hancheng Ye, Peng Ye, Chong Yu, Baopu Li, Tao Chen

TL;DR

This work reveals the relative sparsity effect in emerging stimulative training and proposes a structured pruning framework, named STP, based on an enhanced sparsification paradigm which maintains the magnitude of dropped weights and enhances the expressivity of kept weights by self-distillation.

Abstract

Sparsification-based pruning has been an important category in model compression. Existing methods commonly set sparsity-inducing penalty terms to suppress the importance of dropped weights, which is regarded as the suppressed sparsification paradigm. However, this paradigm inactivates the dropped parts of networks causing capacity damage before pruning, thereby leading to performance degradation. To alleviate this issue, we first study and reveal the relative sparsity effect in emerging stimulative training and then propose a structured pruning framework, named STP, based on an enhanced sparsification paradigm which maintains the magnitude of dropped weights and enhances the expressivity of kept weights by self-distillation. Besides, to find an optimal architecture for the pruned network, we propose a multi-dimension architecture space and a knowledge distillation-guided exploration strategy. To reduce the huge capacity gap of distillation, we propose a subnet mutating expansion technique. Extensive experiments on various benchmarks indicate the effectiveness of STP. Specifically, without fine-tuning, our method consistently achieves superior performance at different budgets, especially under extremely aggressive pruning scenarios, e.g., remaining 95.11% Top-1 accuracy (72.43% in 76.15%) while reducing 85% FLOPs for ResNet-50 on ImageNet. Codes will be released soon.

Enhanced Sparsification via Stimulative Training

TL;DR

This work reveals the relative sparsity effect in emerging stimulative training and proposes a structured pruning framework, named STP, based on an enhanced sparsification paradigm which maintains the magnitude of dropped weights and enhances the expressivity of kept weights by self-distillation.

Abstract

Sparsification-based pruning has been an important category in model compression. Existing methods commonly set sparsity-inducing penalty terms to suppress the importance of dropped weights, which is regarded as the suppressed sparsification paradigm. However, this paradigm inactivates the dropped parts of networks causing capacity damage before pruning, thereby leading to performance degradation. To alleviate this issue, we first study and reveal the relative sparsity effect in emerging stimulative training and then propose a structured pruning framework, named STP, based on an enhanced sparsification paradigm which maintains the magnitude of dropped weights and enhances the expressivity of kept weights by self-distillation. Besides, to find an optimal architecture for the pruned network, we propose a multi-dimension architecture space and a knowledge distillation-guided exploration strategy. To reduce the huge capacity gap of distillation, we propose a subnet mutating expansion technique. Extensive experiments on various benchmarks indicate the effectiveness of STP. Specifically, without fine-tuning, our method consistently achieves superior performance at different budgets, especially under extremely aggressive pruning scenarios, e.g., remaining 95.11% Top-1 accuracy (72.43% in 76.15%) while reducing 85% FLOPs for ResNet-50 on ImageNet. Codes will be released soon.
Paper Structure (16 sections, 11 equations, 11 figures, 9 tables)

This paper contains 16 sections, 11 equations, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sparsification-based Pruning
  4. Knowledge Distillation in Pruning
  5. Stimulative Training
  6. Method
  7. Problem Formutation
  8. Relative Sparsity Effect in Stimulative Training
  9. Framework
  10. Experiment
  11. Image Classifcation
  12. Ablation Studies
  13. Downstream Tasks
  14. Evaluation of Architecture Exploration
  15. Parameter Magnitude Analysis
  16. ...and 1 more sections

Figures (11)

  • Figure 1: ResNet-50 on ImageNet dataset. Top-1 accuracy (%) and remaining FLOPs (in percentage) are reported. Without fine-tuning, our method can still obtain an optimal Pareto frontier of efficiency and performance compared with other methods.
  • Figure 2: Different sparsification methods, namely $L_1$, $L_2$, and ST, on CIFAR-100 dataset under different FLOPs. The subscripts ap and bp represent the performance of networks after and before pruning, respectively.
  • Figure 3: Comparison of suppressed and enhanced sparsification. The exemplified fully connected layer has three input neurons and three output ones. The solid black lines denote the parameters to remain, while the dashed ones are pruned. The parameters connected to a redder neuron have larger magnitudes.
  • Figure 4: Baseline, layer 3-1
  • Figure 5: ST, layer 3-1, 0.3
  • ...and 6 more figures