PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
Max Zimmer, Megi Andoni, Christoph Spiegel, Sebastian Pokutta
TL;DR
The paper tackles the high cost of retraining after pruning large neural networks by showing that updating a tiny, highly expressive subset of parameters (e.g., biases or specific -parameters) can recover or even exceed the performance of full retraining across various sparsity levels. It introduces two LoRA-inspired variants, multlora and masklora, that preserve sparsity while enabling adapters to be merged back into the original weights, and demonstrates memory-efficient layer-wise reconstruction to boost reconstruction-based methods. Empirical results across OPT, LLaMA-2, Mistral, and Mixtral models (up to 30B parameters) show that retraining as little as 0.01–0.05% of parameters can match full retraining, with substantial gains in throughput and memory efficiency. The work offers a practical alternative to full retraining, enabling scalable prune-retrain workflows on very large language models and stimulating further development of parameter-efficient sparse fine-tuning techniques.
Abstract
Neural Networks can be effectively compressed through pruning, significantly reducing storage and compute demands while maintaining predictive performance. Simple yet effective methods like magnitude pruning remove less important parameters and typically require a costly retraining procedure to restore performance. However, with the rise of LLMs, full retraining has become infeasible due to memory and compute constraints. This study challenges the practice of retraining all parameters by showing that updating a small subset of highly expressive parameters can suffice to recover or even enhance performance after pruning. Surprisingly, retraining just 0.01%-0.05% of the parameters in GPT-architectures can match the performance of full retraining across various sparsity levels, significantly reducing compute and memory requirements, and enabling retraining of models with up to 30 billion parameters on a single GPU in minutes. To bridge the gap to full retraining in the high sparsity regime, we introduce two novel LoRA variants that, unlike standard LoRA, allow merging adapters back without compromising sparsity. Going a step further, we show that these methods can be applied for memory-efficient layer-wise reconstruction, significantly enhancing state-of-the-art retraining-free methods like Wanda (Sun et al., 2023) and SparseGPT (Frantar & Alistarh, 2023). Our findings present a promising alternative to avoiding retraining.