SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models

TL;DR

SPP addresses the challenge of retraining post-training pruned LLMs by introducing sparsity-preserved parameter-efficient fine-tuning. It inserts two small learnable matrices with column/row orientations to the sparse weight matrices and applies a residual, element-wise merging to maintain the original sparsity pattern throughout training and weight-merging. Empirical results on LLaMA and LLaMA-2 show that SPP significantly boosts zero-shot performance for both unstructured and N:M sparsity, including up to 75% sparsity, and outperforms prior retraining approaches like LoRA-based variants. The method achieves these gains with a tiny fraction of trainable parameters, preserves inference speedups from sparsity, and offers a practical path to efficient deployment of sparse LLMs.

Abstract

Large Language Models (LLMs) have become pivotal in advancing the field of artificial intelligence, yet their immense sizes pose significant challenges for both fine-tuning and deployment. Current post-training pruning methods, while reducing the sizes of LLMs, often fail to maintain their original performance. To address these challenges, this paper introduces SPP, a Sparsity-Preserved Parameter-efficient fine-tuning method. Different from existing post-training pruning approaches that struggle with performance retention, SPP proposes to employ lightweight learnable column and row matrices to optimize sparse LLM weights, keeping the structure and sparsity of pruned pre-trained models intact. By element-wise multiplication and residual addition, SPP ensures the consistency of model sparsity pattern and ratio during both training and weight-merging processes. We demonstrate the effectiveness of SPP by applying it to the LLaMA and LLaMA-2 model families with recent post-training pruning methods. Our results show that SPP significantly enhances the performance of models with different sparsity patterns (i.e. unstructured and N:M sparsity), especially for those with high sparsity ratios (e.g. 75%), making it a promising solution for the efficient fine-tuning of sparse LLMs. Code will be made available at https://github.com/Lucky-Lance/SPP.

Figures (5)

• Figure 1: We use LLaMA 7B/30B models at 75% sparsity and test zero-shot accuracies on 7 benchmarks of LM-eval gao2021framework to compare different pruning methods. The results of Wanda, Wanda+DS$\oslash$T, Wanda+LoRA*, and Wanda+SPP are visualized. The first two approaches are post-training pruning schemes, LoRA* denotes applying the original Wanda pruning masks to sparse the dense model after LoRA training. Our method achieves overall best results. More experiment details are illustrated in Sec. \ref{['experiments']}.
• Figure 2: Comparison between different methods for sparsification of neural networks. (a) A dense linear layer with matrix $\mathbf{W}^i$ and activation $\mathbf{X}^i$. (b) Post-training pruning methods leverage weight magnitude and calibration data for weight pruning. (c) Model fine-tuning with dynamic weight masks, typically referred to as full fine-tuning methods in the literature, adaptively changes the weight masks during the fine-tuning process. (d) Model fine-tuning with fixed weight masks. Our proposed SPP distinguishes itself as a parameter-efficient fine-tuning algorithm that can consistently maintain model sparsity during both the training and weight-merging phases.
• Figure 3: Learnable parameter insertion of our proposed SPP for a $m\times n$ ($m=4,n=8$) weight matrix. (a) We acquire the mask $\mathbf{M}^i$ of the sparse matrix of a linear layer by pruning with post-training algorithms (e.g. SparseGPT Frantar2023SparseGPTML, Wanda sun2023simple), and obtain sparse linear weights $\widetilde{\mathbf{W}}^i=\mathbf{W}^i\odot \mathbf{M}^i$. (b) SPP adds two sets of parameters to the sparse matrix, one ($\mathbf{W}^i_{\beta}\in \mathbb{R}^{m\times 1}$) initialized to 0 and multiplied on each row of the matrix, and one ($\mathbf{W}^i_{\alpha}\in \mathbb{R}^{r\times n}$, $r=2$ in this example) randomly initialized and multiplied on the columns of the matrix. We then scale $\mathbf{W}^i_{\alpha}$ and $\mathbf{W}^i_{\beta}$ to the size of ($m\times n$) and element-wise multiply then with $\widetilde{\mathbf{W}}^i$.
• Figure 4: Our proposed SPP framework for sparse weight models. We multiply two sets of learnable parameters to the frozen linear weights with $\mathbf{W}_\alpha$ randomly initialized and $\mathbf{W}_\beta$ zero-initialized (detailed in Fig. \ref{['fig:peft-detail']}). Linear operation is conducted on both the original and modified weights, and then the results are added in a residual manner. This framework corresponds to Eq. (\ref{['eq:pipe_sppeft']}).
• Figure 5: The vanilla linear function and the two parallel methods in Megatron-LM shoeybi2019megatron.