Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Class-Aware Pruning for Efficient Neural Networks

Mengnan Jiang, Jingcun Wang, Amro Eldebiky, Xunzhao Yin, Cheng Zhuo, Ing-Chao Lin, Grace Li Zhang

TL;DR

This paper proposes a class-aware pruning technique to compress DNNs, which provides a novel perspective to reduce the computational cost of DNNs and outperforms previous pruning solutions in terms of accuracy, pruning ratio and the reduction of FLOPs.

Abstract

Deep neural networks (DNNs) have demonstrated remarkable success in various fields. However, the large number of floating-point operations (FLOPs) in DNNs poses challenges for their deployment in resource-constrained applications, e.g., edge devices. To address the problem, pruning has been introduced to reduce the computational cost in executing DNNs. Previous pruning strategies are based on weight values, gradient values and activation outputs. Different from previous pruning solutions, in this paper, we propose a class-aware pruning technique to compress DNNs, which provides a novel perspective to reduce the computational cost of DNNs. In each iteration, the neural network training is modified to facilitate the class-aware pruning. Afterwards, the importance of filters with respect to the number of classes is evaluated. The filters that are only important for a few number of classes are removed. The neural network is then retrained to compensate for the incurred accuracy loss. The pruning iterations end until no filter can be removed anymore, indicating that the remaining filters are very important for many classes. This pruning technique outperforms previous pruning solutions in terms of accuracy, pruning ratio and the reduction of FLOPs. Experimental results confirm that this class-aware pruning technique can significantly reduce the number of weights and FLOPs, while maintaining a high inference accuracy.

Class-Aware Pruning for Efficient Neural Networks

TL;DR

This paper proposes a class-aware pruning technique to compress DNNs, which provides a novel perspective to reduce the computational cost of DNNs and outperforms previous pruning solutions in terms of accuracy, pruning ratio and the reduction of FLOPs.

Abstract

Deep neural networks (DNNs) have demonstrated remarkable success in various fields. However, the large number of floating-point operations (FLOPs) in DNNs poses challenges for their deployment in resource-constrained applications, e.g., edge devices. To address the problem, pruning has been introduced to reduce the computational cost in executing DNNs. Previous pruning strategies are based on weight values, gradient values and activation outputs. Different from previous pruning solutions, in this paper, we propose a class-aware pruning technique to compress DNNs, which provides a novel perspective to reduce the computational cost of DNNs. In each iteration, the neural network training is modified to facilitate the class-aware pruning. Afterwards, the importance of filters with respect to the number of classes is evaluated. The filters that are only important for a few number of classes are removed. The neural network is then retrained to compensate for the incurred accuracy loss. The pruning iterations end until no filter can be removed anymore, indicating that the remaining filters are very important for many classes. This pruning technique outperforms previous pruning solutions in terms of accuracy, pruning ratio and the reduction of FLOPs. Experimental results confirm that this class-aware pruning technique can significantly reduce the number of weights and FLOPs, while maintaining a high inference accuracy.
Paper Structure (11 sections, 6 equations, 8 figures, 3 tables)

This paper contains 11 sections, 6 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background
  4. Motivation
  5. Proposed Class-Aware Pruning Method
  6. Modified Neural Network Training
  7. Evaluation of Importance Scores for Filters
  8. Pruning and Fine-Tuning Strategies
  9. The Overall Pruning Framework
  10. Experimental Results
  11. Conclusion

Figures (8)

  • Figure 1: An example of the importance of neurons for different number of classes.
  • Figure 2: Reshaping weights of filters in a convolutional layer to match the flattened input feature map.
  • Figure 3: An example of evaluating the importance score for one activation output $a_{2}^{1}$ in the 1st filter for the $3$rd class, namely cat, where $2$ represents the second activation output. Images ($x_1^{3}, x_2^{3}, \ldots, x_{10}^{3}$) in the $3$rd class are the inputs for the network.
  • Figure 4: Distribution of filter importance scores before and after pruning in single layer. VGG16-CIFAR10: displayed is the first convolutional layer. VGG19-CIFAR100: displayed is the third convolutional layer. ResNet56-CIFAR10/100: displayed is the 40th convolutional layer.
  • Figure 5: The proposed class-aware pruning framework.
  • ...and 3 more figures