Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu

TL;DR

This paper introduces SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework, and proves that statically pruning only the frozen base weights minimizes the pruning error bound.

Abstract

Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs. Magnitude-based pruning can yield sparse models but typically degrades LoRA's performance when applied naively. In this paper, we introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework. We prove that statically pruning only the frozen base weights minimizes the pruning error bound, and we recover the discarded residual information via a truncated-SVD low-rank adapter, which provably reduces per-entry MSE by a factor of $(1 - r/\min(d,k))$. To maximize hardware efficiency, we fuse multiple low-rank adapters into a single concatenated GEMM, and we adopt a bitmap-based encoding with a two-stage pipelined decoding + GEMM design to achieve true model compression and speedup. Empirically, SALR attains 50\% sparsity on various LLMs while matching the performance of LoRA on GSM8K and MMLU, reduces model size by $2\times$, and delivers up to a $1.7\times$ inference speedup.

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

TL;DR

This paper introduces SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework, and proves that statically pruning only the frozen base weights minimizes the pruning error bound.

Abstract

Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs. Magnitude-based pruning can yield sparse models but typically degrades LoRA's performance when applied naively. In this paper, we introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework. We prove that statically pruning only the frozen base weights minimizes the pruning error bound, and we recover the discarded residual information via a truncated-SVD low-rank adapter, which provably reduces per-entry MSE by a factor of . To maximize hardware efficiency, we fuse multiple low-rank adapters into a single concatenated GEMM, and we adopt a bitmap-based encoding with a two-stage pipelined decoding + GEMM design to achieve true model compression and speedup. Empirically, SALR attains 50\% sparsity on various LLMs while matching the performance of LoRA on GSM8K and MMLU, reduces model size by , and delivers up to a inference speedup.
Paper Structure (18 sections, 8 theorems, 74 equations, 3 figures, 7 tables)

This paper contains 18 sections, 8 theorems, 74 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. System Efficiency Limitations in Existing Work
  4. Methodology
  5. Sparsity Preservation Pruning
  6. Concatenating Multi-LoRA adapters
  7. Mapping Sparse Weights and Pipeline Design
  8. Experimental Results
  9. Performance on Domain-Specific Tasks
  10. System Efficiency
  11. Ablation Study
  12. Conclusion
  13. Acknowledgments
  14. Proof of Theoretical Results
  15. Proof of Theorem 1
  16. ...and 3 more sections

Key Result

Theorem 1

Let $W\sim\mathcal{N}(0,\sigma^2)$ and for a given pruning ratio $p\in[0,1)$ we choose the threshold $T_p$ such that Then the mean-squared error (MSE) of this pruning is where $t_p = \Phi^{-1}\bigl(\tfrac{1+p}{2}\bigr)$ and $\varphi(t)=\tfrac{1}{\sqrt{2\pi}}e^{-t^2/2}$ is the standard normal PDF.

Figures (3)

  • Figure 1: Memory-accuracy trade-off on the GSM8K gsm8k benchmark for Llama3-8B llama3, fine-tuned on MetaMath metamath. At 50% sparsity, SALR maintains the dense LoRA lora baseline accuracy (79.5%) while reducing model size from 15.5 GB to 7.98 GB. In contrast, LoSA losa at 50% sparsity suffers a drop to 71.4% accuracy.
  • Figure 2: Overview of SALR. (a) SALR first prunes the base model, resulting in a pruned module and a corresponding residual matrix. Only the bitmap and the dense entries of the pruned module are stored. Subsequently, SALR decomposes the residual into a single low-rank pair using the optimal rank-$r$ approximation. (b) Bitmap decoding of sparse weights.
  • Figure 3: Normalized cumulative singular-value energy spectra of the residual correction matrices for LoSA and SALR on Llama3-8B after fine-tuning on MetaMath.

Theorems & Definitions (12)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem
  • proof
  • Theorem
  • proof
  • Theorem
  • proof
  • ...and 2 more