Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples

Shiva Sreeram, Alaa Maalouf, Pratyusha Sharma, Daniela Rus

TL;DR

This work addresses the costly problem of adapting large language models to new domains without gradient-based fine-tuning. It introduces a gradient-guided, sample-efficient pipeline that identifies which weight matrices to compress via singular-value gradients, expands the factorization space with multi-subspace (block) SVDs, and uses a 100-example evaluation to select final compressions. The approach yields substantial practical benefits, achieving up to 52× speedups and up to 24.6 percentage-point accuracy gains across eight benchmarks and two model families (GPT-J and RoBERTa) while avoiding fine-tuning. Overall, it enables training-free, robust LLM adaptation suitable for resource-constrained deployment, with potential extensions to multilingual and retrieval-augmented setups in future work.

Abstract

Recently, Sharma et al. suggested a method called Layer-SElective-Rank reduction (LASER) which demonstrated that pruning high-order components of carefully chosen LLM's weight matrices can boost downstream accuracy -- without any gradient-based fine-tuning. Yet LASER's exhaustive, per-matrix search (each requiring full-dataset forward passes) makes it impractical for rapid deployment. We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected -- eliminating the layer-by-layer sweep, (ii) The gradient of each matrix's singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data -- both for computing the indicative gradients and for measuring the final accuracy -- suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size. As a result, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets -- entirely without fine-tuning.

Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples

TL;DR

This work addresses the costly problem of adapting large language models to new domains without gradient-based fine-tuning. It introduces a gradient-guided, sample-efficient pipeline that identifies which weight matrices to compress via singular-value gradients, expands the factorization space with multi-subspace (block) SVDs, and uses a 100-example evaluation to select final compressions. The approach yields substantial practical benefits, achieving up to 52× speedups and up to 24.6 percentage-point accuracy gains across eight benchmarks and two model families (GPT-J and RoBERTa) while avoiding fine-tuning. Overall, it enables training-free, robust LLM adaptation suitable for resource-constrained deployment, with potential extensions to multilingual and retrieval-augmented setups in future work.

Abstract

Recently, Sharma et al. suggested a method called Layer-SElective-Rank reduction (LASER) which demonstrated that pruning high-order components of carefully chosen LLM's weight matrices can boost downstream accuracy -- without any gradient-based fine-tuning. Yet LASER's exhaustive, per-matrix search (each requiring full-dataset forward passes) makes it impractical for rapid deployment. We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected -- eliminating the layer-by-layer sweep, (ii) The gradient of each matrix's singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data -- both for computing the indicative gradients and for measuring the final accuracy -- suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size. As a result, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets -- entirely without fine-tuning.
Paper Structure (23 sections, 1 theorem, 18 equations, 4 figures, 9 tables, 1 algorithm)

This paper contains 23 sections, 1 theorem, 18 equations, 4 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Gradient of a Singular Value w.r.t. the Loss
  5. A tiny set is enough for evaluation and gradient calculation
  6. Denoising LLMs layers with multiple subspaces/SVDs factorizations
  7. Experimental Results
  8. Improving upon the state of the art (SOTA)
  9. Ablation of the proposed efficiency improvement techniques
  10. Ablation on single subspace vs multi-subspace rank reduction with LASER
  11. Ablating the effect of optimized clustering in LASER
  12. Conclusion and Future Work
  13. Technical Appendices and Supplementary Material
  14. Specific compute times
  15. Parameters for Tables \ref{['tab:gptj-clustering-eff']} and \ref{['tab:roberta-clustering-eff']}
  16. ...and 8 more sections

Key Result

Lemma 1

Let $W \in \mathbb{R}^{m\times n}$ have rank $r \le \min\{m,n\}$ and a unique SVD $W = U \operatorname{diag}(\sigma_1,\dots,\sigma_r) V^\top$ with orthonormal columns $U = [u_1,\dots,u_r]\in\mathbb{R}^{m\times r}$ and $V = [v_1,\dots,v_r]\in\mathbb{R}^{n\times r}$, whose singular values satisfy $\si where $\operatorname{diag}(\cdot)$ extracts the diagonal of its square argument.

Figures (4)

  • Figure 1: Efficient LLM adaptation. We present a method to adapt LLMs to new styles/domains without fine-tuning. (1) With a single gradient step on the target data, we compute gradients of the singular values across all weight matrices; these gradients rank which matrices merit low-rank compression to curb overfitting and align to the new style. (2) We broaden the search by clustering the rows of the selected matrices into multiple (best fitting) subspaces and factor each cluster, capturing heterogeneous structure and reducing noise/overfitting that manifest differently across row groups. (3) We show that both the gradient scoring and evaluation can be done with just 100 examples. (4) Finally, this yields up to 52× speedups and up to +24.6-point accuracy gains—no fine-tuning required.
  • Figure 2: The accuracy of techniques given computation time for eight datasets while running with GPT-J. Line drawn between Baseline and LASER points to highlight ratio of accuracy and compute.
  • Figure 3: Comparison of singular values versus gradient diagonal values for the optimal matrix result on GPT-J FEVER. The correlation statistics are also provided.
  • Figure : Block-First Gradient Low-Rank Adaptation

Theorems & Definitions (2)

  • Lemma 1: Gradient w.r.t. a singular value
  • proof