Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

Jianbo Ye, Xin Lu, Zhe Lin, James Z. Wang

TL;DR

This work challenges the conventional smaller-norm informativeness assumption in channel pruning and introduces a BN γ-based pruning approach. By applying end-to-end ISTA updates to γ and a γ–W rescaling trick, the method aggressively prunes channels while preserving network functionality, absorbing constant channels into subsequent biases. It demonstrates strong empirical results across CIFAR-10, ImageNet (ResNet-101), and segmentation tasks, achieving substantial parameter and FLOP reductions with competitive accuracy. The approach is simple to reproduce, hardware-friendly, and applicable to BN-enabled convolutional architectures, offering a practical path to faster, leaner CNNs.

Abstract

Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios. A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-to- end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interesting aspects and competitive performance.

Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

TL;DR

This work challenges the conventional smaller-norm informativeness assumption in channel pruning and introduces a BN γ-based pruning approach. By applying end-to-end ISTA updates to γ and a γ–W rescaling trick, the method aggressively prunes channels while preserving network functionality, absorbing constant channels into subsequent biases. It demonstrates strong empirical results across CIFAR-10, ImageNet (ResNet-101), and segmentation tasks, achieving substantial parameter and FLOP reductions with competitive accuracy. The approach is simple to reproduce, hardware-friendly, and applicable to BN-enabled convolutional architectures, offering a practical path to faster, leaner CNNs.

Abstract

Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios. A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-to- end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interesting aspects and competitive performance.

Paper Structure

This paper contains 12 sections, 7 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Rethinking the Smaller-Norm-Less-Informative Assumption
  4. Channel Pruning of Batch-Normalized CNN
  5. Preliminaries
  6. The Algorithm
  7. Guidelines for Tuning Hyper-parameters
  8. Experiments
  9. CIFAR-10 Experiment
  10. ILSVRC2012 Experiment
  11. Image Foreground-Background Segmentation Experiment
  12. Conclusions

Figures (1)

  • Figure 1: Visualization of the number of pruned channels at each convolution in the inception branch. Colored regions represents the number of channels kept. The height of each bar represents the size of feature map, and the width of each bar represents the size of channels. It is observed that most of channels in the bottom layers are kept while most of channels in the top layers are pruned.