Information Consistent Pruning: How to Efficiently Search for Sparse Networks?

TL;DR

This work tackles the slow training cycle of Iterative Magnitude Pruning by introducing Information Consistent Pruning (InCoP), a flow-based stopping criterion that uses layerwise Information Flow $\Delta$ and Gradient Flow $\mathbf{g}$ to monitor proximity to the optimally dense network. Building on SAP, InCoP adds adaptive pruning with a principled stopping rule, yielding two variants InCoP-IF and InCoP-GF that can significantly reduce retraining epochs while achieving the same final accuracy as traditional IMPs. The authors provide theoretical insights and proofs, and demonstrate through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 with ResNet and VGG architectures that InCoP improves time efficiency across dataset-model pairs. The approach offers practical impact for deploying sparse networks on resource-constrained devices by lowering training time without sacrificing final performance, with tool availability at the authors' repository and avenues for extending flow-based proximity metrics.

Abstract

Iterative magnitude pruning methods (IMPs), proven to be successful in reducing the number of insignificant nodes in over-parameterized deep neural networks (DNNs), have been getting an enormous amount of attention with the rapid deployment of DNNs into cutting-edge technologies with computation and memory constraints. Despite IMPs popularity in pruning networks, a fundamental limitation of existing IMP algorithms is the significant training time required for each pruning iteration. Our paper introduces a novel \textit{stopping criterion} for IMPs that monitors information and gradient flows between networks layers and minimizes the training time. Information Consistent Pruning (\ourmethod{}) eliminates the need to retrain the network to its original performance during intermediate steps while maintaining overall performance at the end of the pruning process. Through our experiments, we demonstrate that our algorithm is more efficient than current IMPs across multiple dataset-DNN combinations. We also provide theoretical insights into the core idea of our algorithm alongside mathematical explanations of flow-based IMP. Our code is available at \url{https://github.com/Sekeh-Lab/InfCoP}.

Figures (7)

• Figure 1: a. Accuracy-based stopping criterion: the overall performances of the optimal and sparse networks are compared b. gradient flow-based: the gradient flow of corresponding layers are compared (similar shades of colors are compared between the two networks) and c. information flow-based: the information flow of corresponding consecutive layers are compared (similarly, boxes with similar shades of colors are compared).
• Figure 2: SGD in pruning with accuracy based training vs. flow based training. The red region corresponds to the set $S$, representing the information based stopping criterion, while the green region corresponds to the set $M$, symbolizing the accuracy based stopping criterion. The shift and contraction of these areas occur due to dimensionality reduction resulting from the pruning step.
• Figure 3: ResNet18 - MNIST (top), FashionMNIST (middle), CIFAR10 (bottom). The left column shows accuracy at each iteration $t$, the number of training epochs required in $t$ is shown in middle column, and the right column shows remaining weights of $F_w^{(L)}$ in different iterations. Purple line $\rightarrow (p, q) = (1, 2)$, green $\rightarrow (p, q)=(0.5, 1)$, and cyan $\rightarrow$ LTH.
• Figure 4: VGG16 - MNIST (top), FashionMNIST (middle), CIFAR10 (bottom). As for Fig. \ref{['fig:resnet']} purple line indicates $(p, q) = (1, 2)$, green is $(p, q)=(0.5, 1)$, and LTH is cyan line.
• Figure 5: Comparison of execution times for a single IMP experiment across different pruning methods.

Theorems & Definitions (5)

• lemma thmcounterlemma
• proof
• lemma thmcounterlemma
• proof
• lemma thmcounterlemma