FGGP: Fixed-Rate Gradient-First Gradual Pruning

TL;DR

The proposed fixed-rate gradient-first gradual pruning (FGGP) approach outperforms its state-of-the-art alternatives in most of the above experimental settings, even occasionally surpassing the upperbound of corresponding dense network results, and having the highest ranking across the considered experimental settings.

Abstract

In recent years, the increasing size of deep learning models and their growing demand for computational resources have drawn significant attention to the practice of pruning neural networks, while aiming to preserve their accuracy. In unstructured gradual pruning, which sparsifies a network by gradually removing individual network parameters until a targeted network sparsity is reached, recent works show that both gradient and weight magnitudes should be considered. In this work, we show that such mechanism, e.g., the order of prioritization and selection criteria, is essential. We introduce a gradient-first magnitude-next strategy for choosing the parameters to prune, and show that a fixed-rate subselection criterion between these steps works better, in contrast to the annealing approach in the literature. We validate this on CIFAR-10 dataset, with multiple randomized initializations on both VGG-19 and ResNet-50 network backbones, for pruning targets of 90, 95, and 98% sparsity and for both initially dense and 50% sparse networks. Our proposed fixed-rate gradient-first gradual pruning (FGGP) approach outperforms its state-of-the-art alternatives in most of the above experimental settings, even occasionally surpassing the upperbound of corresponding dense network results, and having the highest ranking across the considered experimental settings.

Figures (3)

• Figure 1: Comparison of the parameter selection mechanisms between our proposed FGGP and GraNet liu2021neuroreg.
• Figure 2: (a) Comparison of FGGP for four different subset selection ratios $r$. (b) Ablations of FGGP indicated as ($\cdot$,$\cdot$) where the first indicated the pruning criteria order ($g$$\rightarrow$$\theta$: gradient-first & $\theta$$\rightarrow$$g$: magnitude-first) and the second the subset selection strategy (with the rate fixed or varying as cosine annealed). GraNet's proposed method choices are also indicated in the same notation for clarity. Note that FGGP($g$$\rightarrow$$\theta$, fixed) is our proposed mechanism with gradient-first fixed-rate subset selection. Experiments are reported for dense-to-sparse pruning of VGG-19 for target sparsities of {90,95,98}%.
• Figure 3: (a) Number of parameters per layer in a dense and FGGP-pruned VGG-19 network, shown in logarithmic scale. (b) Sparsity of each layer after pruning. The results are shown for a sample experiment. Note that the final layer is fully-connected while the others are convolutional.