Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NOLA: Compressing LoRA using Linear Combination of Random Basis

Soroush Abbasi Koohpayegani, KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, Hamed Pirsiavash

TL;DR

NOLA addresses the storage and adaptation challenges of fine-tuning large models by introducing a random-basis reparameterization that decouples trainable parameters from rank and architecture. It rewrites the LoRA update as $\Delta W = AB$ with $A = \sum_{i=1}^k \alpha_i A_i$ and $B = \sum_{j=1}^l \beta_j B_j$, learning only $\alpha$ and $\beta$ while freezing the random bases; seeds and coefficients are stored for reconstruction. This approach yields compression beyond LoRA's rank-one bound and proves effective across GPT-2, LLaMA-2, and Vision Transformers, achieving up to ~20x compression on LLaMA-2 70B without accuracy loss and enabling quantization-friendly training. Quantization of coefficients (e.g., 4-bit) further reduces storage with minimal performance impact, and the method scales to CNNs and GPU-friendly on-the-fly basis generation, making it practical for rapid task-switching in production systems.

Abstract

Fine-tuning Large Language Models (LLMs) and storing them for each downstream task or domain is impractical because of the massive model size (e.g., 350GB in GPT-3). Current literature, such as LoRA, showcases the potential of low-rank modifications to the original weights of an LLM, enabling efficient adaptation and storage for task-specific models. These methods can reduce the number of parameters needed to fine-tune an LLM by several orders of magnitude. Yet, these methods face two primary limitations: (1) the parameter count is lower-bounded by the rank one decomposition, and (2) the extent of reduction is heavily influenced by both the model architecture and the chosen rank. We introduce NOLA, which overcomes the rank one lower bound present in LoRA. It achieves this by re-parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear mixture coefficients only. This approach allows us to decouple the number of trainable parameters from both the choice of rank and the network architecture. We present adaptation results using GPT-2, LLaMA-2, and ViT in natural language and computer vision tasks. NOLA performs as well as LoRA models with much fewer number of parameters compared to LoRA with rank one, the best compression LoRA can archive. Particularly, on LLaMA-2 70B, our method is almost 20 times more compact than the most compressed LoRA without degradation in accuracy. Our code is available here: https://github.com/UCDvision/NOLA

NOLA: Compressing LoRA using Linear Combination of Random Basis

TL;DR

NOLA addresses the storage and adaptation challenges of fine-tuning large models by introducing a random-basis reparameterization that decouples trainable parameters from rank and architecture. It rewrites the LoRA update as with and , learning only and while freezing the random bases; seeds and coefficients are stored for reconstruction. This approach yields compression beyond LoRA's rank-one bound and proves effective across GPT-2, LLaMA-2, and Vision Transformers, achieving up to ~20x compression on LLaMA-2 70B without accuracy loss and enabling quantization-friendly training. Quantization of coefficients (e.g., 4-bit) further reduces storage with minimal performance impact, and the method scales to CNNs and GPU-friendly on-the-fly basis generation, making it practical for rapid task-switching in production systems.

Abstract

Fine-tuning Large Language Models (LLMs) and storing them for each downstream task or domain is impractical because of the massive model size (e.g., 350GB in GPT-3). Current literature, such as LoRA, showcases the potential of low-rank modifications to the original weights of an LLM, enabling efficient adaptation and storage for task-specific models. These methods can reduce the number of parameters needed to fine-tune an LLM by several orders of magnitude. Yet, these methods face two primary limitations: (1) the parameter count is lower-bounded by the rank one decomposition, and (2) the extent of reduction is heavily influenced by both the model architecture and the chosen rank. We introduce NOLA, which overcomes the rank one lower bound present in LoRA. It achieves this by re-parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear mixture coefficients only. This approach allows us to decouple the number of trainable parameters from both the choice of rank and the network architecture. We present adaptation results using GPT-2, LLaMA-2, and ViT in natural language and computer vision tasks. NOLA performs as well as LoRA models with much fewer number of parameters compared to LoRA with rank one, the best compression LoRA can archive. Particularly, on LLaMA-2 70B, our method is almost 20 times more compact than the most compressed LoRA without degradation in accuracy. Our code is available here: https://github.com/UCDvision/NOLA
Paper Structure (13 sections, 2 equations, 2 figures, 11 tables)

This paper contains 13 sections, 2 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Proposed Method: NOLA
  3. Experiments
  4. NOLA on GPT-2:
  5. NOLA with Quantized Coefficients, $\alpha$ and $\beta$:
  6. NOLA on LLaMA-2:
  7. NOLA on Vision Transformers
  8. Related Works
  9. Appendix
  10. Measuring the rank of possible solutions in NOLA vs PRANC:
  11. NOLA in training from scratch:
  12. Ablation and details of NOLA on Vision Transformers:
  13. Results of NLG task on DART and WebNLG datasets:

Figures (2)

  • Figure 1: Our Method (NOLA): After constraining the rank of $\Delta W$ by decomposing it to $A \times B$, we reparametrize $A$ and $B$ to be a linear combination of several random basis matrices. We freeze the basis and $W$ and learn the combination coefficients. To reconstruct the model, we store the coefficients and the seed of the random generator which is a single scalar. NOLA results in more compression compared to LoRA and more importantly decouples the compression ratio from the rank and dimensions of $W$. One can reduce the number of parameters to 4 times smaller than rank=1 of LoRA which is not possible with LoRA due to rank being an integer number.
  • Figure 2: Comparing the rank of samples in the solution subspace for PRANC and NOLA, given the same number of parameters, $n$. "Percentage Coverage" is the subspace rank divided by the max possible rank ($d^2$), so $1.0$ denotes full rank. As expected, the coverage for PRANC increases linearly while it saturates very fast for NOLA.