Reconstruct the Pruned Model without Any Retraining

TL;DR

The paper tackles the practicality bottleneck of retraining-based pruning for large language models by proposing LIAR, a retraining-free, generalizable distortion-reconstruction framework. LIAR reframes pruning distortion as a reconstruction problem and uses LS-estimated transformation matrices $\mathbf{Q}^{\ell}$ and $\mathbf{P}^{\ell}$ to linearly interpolate preserved weights and biases, yielding updated outputs that closely match the unpruned model. Empirically, LIAR preserves up to 98% of BERT-BASE accuracy with 50% pruning and delivers top performance for LLaMA models within minutes, while demonstrating strong generalization across pruning modules and criteria and requiring only a small calibration dataset. This approach enables efficient, hardware-friendly pruning suitable for encoder- and decoder-based transformers, offering a practical path to deployment of compressed LLMs without retraining.

Abstract

Structured pruning is a promising hardware-friendly compression technique for large language models (LLMs), which is expected to be retraining-free to avoid the enormous retraining cost. This retraining-free paradigm involves (1) pruning criteria to define the architecture and (2) distortion reconstruction to restore performance. However, existing methods often emphasize pruning criteria while using reconstruction techniques that are specific to certain modules or criteria, resulting in limited generalizability. To address this, we introduce the Linear Interpolation-based Adaptive Reconstruction (LIAR) framework, which is both efficient and effective. LIAR does not require back-propagation or retraining and is compatible with various pruning criteria and modules. By applying linear interpolation to the preserved weights, LIAR minimizes reconstruction error and effectively reconstructs the pruned output. Our evaluations on benchmarks such as GLUE, SQuAD, WikiText, and common sense reasoning show that LIAR enables a BERT model to maintain 98% accuracy even after removing 50% of its parameters and achieves top performance for LLaMA in just a few minutes.

Figures (9)

• Figure 1: (a) Accuracy drop on the STS-B task by dropping 70% FFN neurons of BERT with various pruning criteria (x-axis) and reconstruction methods (legends). '*' means the retraining-based criteria. (b) Reconstruction error across different tokens and samples by Bias Compensation and LIAR.
• Figure 2: Overview of LIAR framework. The original output is first reformulated by (1) Reconstruction Problem Reformulation, and then the masked output is estimated by (2) Least Square-based Linear Estimation. Finally, the model weight and bias are updated with (3) Linear Interpolation.
• Figure 3: Accuracy comparison of BERT$_{\rm BASE}$ on STS-B, MNLI, QQP and SQuAD$_{1.1}$ tasks by pruning attention heads (upper) and FFN neurons (lower) with different reconstruction strategies.
• Figure 4: Accuracy comparison of the BERT$_{\rm BASE}$ pruned by Weight Magnitude-based (upper) and SNIP (lower) criteria on the STS-B, MNLI, SST-2 and SQuAD$_{1.1}$ tasks respectively. We only prune the FFN neurons to avoid introducing the architecture search problem.
• Figure 5: (a) Reconstruction error distribution of the hidden input of LLaMA-7B across 1024 instances sampled from the WikiText-2 traning dataset. (b) Perplexity comparison of the LLaMA-7B by removing the updation for the bias and weight term at various pruning ratios.