Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Quest for Winning Tickets in Low-Rank Adapters

Hamed Damirchi, Cristian Rodriguez-Opazo, Ehsan Abbasnejad, Zhen Zhang, Javen Shi

TL;DR

This work extends the Lottery Ticket Hypothesis to Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning, showing that sparsely masked LoRAs can match dense LoRA performance while dramatically reducing trainable parameters. It introduces Partial-LoRA, which derives per-layer sparsity ratios from limited data and applies random masks to LoRA components, preserving task-relevant subspaces of pretrained models. Theoretical guarantees are provided for the existence of winning tickets in LoRAs, supported by empirical results across 8 vision and 12 language tasks, larger models, and multi-task settings, achieving up to 87% parameter reduction with competitive accuracy. This approach offers a principled, scalable path to efficient adaptation of large pretrained models with broad practical impact in real-world deployment where resources are limited.

Abstract

The Lottery Ticket Hypothesis (LTH) suggests that over-parameterized neural networks contain sparse subnetworks ("winning tickets") capable of matching full model performance when trained from scratch. With the growing reliance on fine-tuning large pretrained models, we investigate whether LTH extends to parameter-efficient fine-tuning (PEFT), specifically focusing on Low-Rank Adaptation (LoRA) methods. Our key finding is that LTH holds within LoRAs, revealing sparse subnetworks that can match the performance of dense adapters. In particular, we find that the effectiveness of sparse subnetworks depends more on how much sparsity is applied in each layer than on the exact weights included in the subnetwork. Building on this insight, we propose Partial-LoRA, a method that systematically identifies said subnetworks and trains sparse low-rank adapters aligned with task-relevant subspaces of the pre-trained model. Experiments across 8 vision and 12 language tasks in both single-task and multi-task settings show that Partial-LoRA reduces the number of trainable parameters by up to 87\%, while maintaining or improving accuracy. Our results not only deepen our theoretical understanding of transfer learning and the interplay between pretraining and fine-tuning but also open new avenues for developing more efficient adaptation strategies.

The Quest for Winning Tickets in Low-Rank Adapters

TL;DR

This work extends the Lottery Ticket Hypothesis to Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning, showing that sparsely masked LoRAs can match dense LoRA performance while dramatically reducing trainable parameters. It introduces Partial-LoRA, which derives per-layer sparsity ratios from limited data and applies random masks to LoRA components, preserving task-relevant subspaces of pretrained models. Theoretical guarantees are provided for the existence of winning tickets in LoRAs, supported by empirical results across 8 vision and 12 language tasks, larger models, and multi-task settings, achieving up to 87% parameter reduction with competitive accuracy. This approach offers a principled, scalable path to efficient adaptation of large pretrained models with broad practical impact in real-world deployment where resources are limited.

Abstract

The Lottery Ticket Hypothesis (LTH) suggests that over-parameterized neural networks contain sparse subnetworks ("winning tickets") capable of matching full model performance when trained from scratch. With the growing reliance on fine-tuning large pretrained models, we investigate whether LTH extends to parameter-efficient fine-tuning (PEFT), specifically focusing on Low-Rank Adaptation (LoRA) methods. Our key finding is that LTH holds within LoRAs, revealing sparse subnetworks that can match the performance of dense adapters. In particular, we find that the effectiveness of sparse subnetworks depends more on how much sparsity is applied in each layer than on the exact weights included in the subnetwork. Building on this insight, we propose Partial-LoRA, a method that systematically identifies said subnetworks and trains sparse low-rank adapters aligned with task-relevant subspaces of the pre-trained model. Experiments across 8 vision and 12 language tasks in both single-task and multi-task settings show that Partial-LoRA reduces the number of trainable parameters by up to 87\%, while maintaining or improving accuracy. Our results not only deepen our theoretical understanding of transfer learning and the interplay between pretraining and fine-tuning but also open new avenues for developing more efficient adaptation strategies.
Paper Structure (30 sections, 2 theorems, 13 equations, 17 figures, 8 tables, 1 algorithm)

This paper contains 30 sections, 2 theorems, 13 equations, 17 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Methodology
  5. Existence of Lottery Tickets in LoRA Residuals
  6. Finding Subnetworks in Pretrained Models
  7. Deriving Sparsity Ratios
  8. Importance Measures
  9. Experiments
  10. Sparsifying LoRAs
  11. Comparison to the SoTA
  12. Sweeping through sparsity ratios
  13. Larger Models
  14. Sparsity Ratio Derivation Ablation
  15. Ablation On The LoRA Rank
  16. ...and 15 more sections

Key Result

Theorem 4.1

Define a network $f_{T}$ of depth $L$, parameterized by pretrained weights and biases $W_l$ and $b_l$, with low-rank adapters at each layer parameterized by residuals $\Delta W^l_T$. Additionally, define a pruned network $f_{LoRA}$ of depth $L+1$, parameterized by $\Delta W^l \cdot U$, where $U\sim

Figures (17)

  • Figure 1: The process of sparsifying LoRAs (left) involves extracting sparsity ratios from the pre-trained model to generate random masks for low-rank adapter components, yielding outputs comparable to fully-parameterized LoRA while substantially reducing trainable parameters, as demonstrated for vision and language tasks (right).
  • Figure 2: Change in accuracy is visualized against parameter count (in millions) after the application of our sparsification method for both vision and language models. Both Partial-AdaLoRA and Partial-LoRA performance remains consistent after significant sparsification.
  • Figure 3: Despite the fewer parameters of our Partial method, our results are competitive with SotA adaptation methods on vision datasets with improvements in datasets prone to overfitting (Flowers).
  • Figure 4: Partial-LoRA and Partial-AdaLoRA stay competitive with SotA methods on language datasets despite the low parameter count.
  • Figure 5: Sweeping sparsity ratio shows sparsification does not lower performance. Partial-LoRA adjusts parameter count to meet dataset demands.
  • ...and 12 more figures

Theorems & Definitions (2)

  • Theorem 4.1
  • Theorem 4.1