Comparing Rewinding and Fine-tuning in Neural Network Pruning
Alex Renda, Jonathan Frankle, Michael Carbin
TL;DR
<3-5 sentence high-level summary> The paper addresses how retraining after pruning should be conducted to maximize parameter efficiency without sacrificing accuracy. It compares three retraining approaches—fine-tuning, weight rewinding, and learning rate rewinding—across unstructured and structured pruning on multiple domains, introducing learning rate rewinding as a network-agnostic, hyperparameter-friendly alternative. The key findings show that both rewinding methods outperform fine-tuning, with learning rate rewinding often achieving the best accuracy and state-of-the-art tradeoffs when combined with iterative pruning. The work provides a practical pruning algorithm and links its results to the Lottery Ticket Hypothesis, offering a strong baseline for future pruning research and real-world deployment.</output>
Abstract
Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, we compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al., (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.