Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation

Tianyi Zhang, Junda Su, Aditya Desai, Oscar Wu, Zhaozhuo Xu, Anshumali Shrivastava

TL;DR

The work tackles the resource bottlenecks of fine-tuning large language models by challenging low-rank adapter assumptions and quantization-driven approaches. It introduces SketchTune, a learned weight-sketching framework that compresses entire weight matrices into small, differentiable sketches and a fixed mapping, enabling direct adaptation with far fewer trainable parameters and no dual-path computation. Theoretical analysis supports the superiority of sketching over low-rank methods for broad classes of weight updates, while extensive experiments across Llama, Llama-2, Llama-3, and Mistral demonstrate superior accuracy with much smaller base models and competitive efficiency. Practical CUDA implementations and results on math, commonsense, and instruction tasks underscore SketchTune as a scalable, efficient pathway for deploying and adapting large language models.

Abstract

Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size. Parameter-efficient fine-tuning (PEFT) techniques typically employ additive adapters applied to frozen model weights. To further reduce memory usage, model weights are often compressed through quantization. However, existing PEFT methods often yield suboptimal model quality because they rely on restrictive assumptions, such as low-rank constraints on adapters to limit the number of trainable parameters. We find that sketching, a popular data compression technique, can serve as an efficient LLM adaptation strategy while avoiding the low-rank assumption. We introduce SketchTune, a compressive adaptation strategy that compresses LLM weights into compact fine-tunable sketches, integrating compression and adaptation into a unified framework. This integration eliminates the need for complex two-path computation in existing PEFT techniques, enabling faster and more memory-efficient training and inference. SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods. Our extensive evaluations with Llama and Mistral models demonstrate that SketchTune outperforms leading PEFT methods across diverse tasks while using substantially smaller base models and comparable trainable parameters. As a highlight, SketchTune outperforms LoRA, DoRA, and S2FT on commonsense and math benchmarks using 2.6-3.5$\times$ smaller base models and exceeds LoftQ in accuracy by 14.48% on GSM8K with 7.3$\times$ fewer trainable parameters. Our code is available at https://github.com/LeanModels/SketchTune.

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation

TL;DR

The work tackles the resource bottlenecks of fine-tuning large language models by challenging low-rank adapter assumptions and quantization-driven approaches. It introduces SketchTune, a learned weight-sketching framework that compresses entire weight matrices into small, differentiable sketches and a fixed mapping, enabling direct adaptation with far fewer trainable parameters and no dual-path computation. Theoretical analysis supports the superiority of sketching over low-rank methods for broad classes of weight updates, while extensive experiments across Llama, Llama-2, Llama-3, and Mistral demonstrate superior accuracy with much smaller base models and competitive efficiency. Practical CUDA implementations and results on math, commonsense, and instruction tasks underscore SketchTune as a scalable, efficient pathway for deploying and adapting large language models.

Abstract

Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size. Parameter-efficient fine-tuning (PEFT) techniques typically employ additive adapters applied to frozen model weights. To further reduce memory usage, model weights are often compressed through quantization. However, existing PEFT methods often yield suboptimal model quality because they rely on restrictive assumptions, such as low-rank constraints on adapters to limit the number of trainable parameters. We find that sketching, a popular data compression technique, can serve as an efficient LLM adaptation strategy while avoiding the low-rank assumption. We introduce SketchTune, a compressive adaptation strategy that compresses LLM weights into compact fine-tunable sketches, integrating compression and adaptation into a unified framework. This integration eliminates the need for complex two-path computation in existing PEFT techniques, enabling faster and more memory-efficient training and inference. SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods. Our extensive evaluations with Llama and Mistral models demonstrate that SketchTune outperforms leading PEFT methods across diverse tasks while using substantially smaller base models and comparable trainable parameters. As a highlight, SketchTune outperforms LoRA, DoRA, and S2FT on commonsense and math benchmarks using 2.6-3.5 smaller base models and exceeds LoftQ in accuracy by 14.48% on GSM8K with 7.3 fewer trainable parameters. Our code is available at https://github.com/LeanModels/SketchTune.
Paper Structure (35 sections, 2 theorems, 41 equations, 4 figures, 13 tables, 1 algorithm)

This paper contains 35 sections, 2 theorems, 41 equations, 4 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. SketchTune: Fine-Tunable Sketches for LLM Adaptation
  3. Weight Updates Are Far from Low-Rank: Sketching Provides a Superior Alternative
  4. Formulation: Learning to Sketch LLM Weights
  5. A Row-Wise Learning Strategy
  6. Scaling Up Number of Trainable Parameters
  7. Fine-Tuning Sketches
  8. Custom CUDA Kernel for Efficient Training and Inference
  9. Theoretical Analysis
  10. Experiments
  11. Results
  12. Comparison with PEFT Methods
  13. Comparison with Compressive Fine-Tuning
  14. Efficiency Analysis
  15. Related Works
  16. ...and 20 more sections

Key Result

Theorem 3.1

Consider a matrix $\mathbf \Delta: n \times n$ with sorted (descending) singular values $\{\rho_i\}_{i=1}^n$, squares of which are drawn from power law $i^{-\eta}$ parameterized by coefficient $\eta$. Under the compression factor $\alpha$ (i.e. using $n^2/\alpha$ parameters), let low-rank approximat The expected error of random-fold sketching approximation is, For large enough $n$, the expected s

Figures (4)

  • Figure 1: (Top 2) Minimum rank required by low-rank matrices to explain varying percentages of variance in fine-tuned LLM weight updates. (Bottom 4) Optimal approximation errors for sketching and low-rank matrices under different compression ratios.
  • Figure 2: An illustration of SketchTune’s process of sketching for model compression and fine-tuning.
  • Figure 3: A comparison on the training and inference efficiency of SketchTune, LoRA, and QLoRA.
  • Figure 4: Depending on the power-law coefficient in the singular values (a soft proxy for rank) of the unknown $\Delta$ matrix, one method dominates the other. We show this classification for a sample case of $n=1024$ and $k=2$

Theorems & Definitions (2)

  • Theorem 3.1
  • Theorem 6.1