Beyond One-Way Pruning: Bidirectional Pruning-Regrowth for Extreme Accuracy-Sparsity Tradeoff
Junchen Liu, Yi Sheng
TL;DR
This paper addresses the sharp performance degradation that accompanies extreme sparsity in neural networks when deployed on resource-constrained devices. It introduces bidirectional pruning-regrowth, a strategy that starts from an already highly sparse model and selectively regrows critical connections to recover lost accuracy, counteracting the typical pruning-induced drop. The authors compare conventional iterative and one-shot pruning and demonstrate that regrowth substantially recovers accuracy, enabling higher sparsity at negligible performance loss, with evidence on VGG16 showing recovery from 91.73 at 98.85% sparsity to 92.64 at 96.85% sparsity for iterative regrowth and to 92.77/92.90 at 96.39% and 92.96% sparsity for one-shot regrowth. They also discuss limitations of existing pruning methods such as Pruning On-the-fly, HRank, and ResRep, arguing that regrowth avoids permanently zeroed weights and excessive hyperparameter tuning. The approach holds potential for deploying extreme-compression models on edge hardware and could extend to large-scale models like LLMs.
Abstract
As a widely adopted model compression technique, model pruning has demonstrated strong effectiveness across various architectures. However, we observe that when sparsity exceeds a certain threshold, both iterative and one-shot pruning methods lead to a steep decline in model performance. This rapid degradation limits the achievable compression ratio and prevents models from meeting the stringent size constraints required by certain hardware platforms, rendering them inoperable. To overcome this limitation, we propose a bidirectional pruning-regrowth strategy. Starting from an extremely compressed network that satisfies hardware constraints, the method selectively regenerates critical connections to recover lost performance, effectively mitigating the sharp accuracy drop commonly observed under high sparsity conditions.