Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression

Ayush Kaushal, Tejas Vaidhya, Irina Rish

TL;DR

The paper proposes Low Rank Decomposition (LoRD) as a one-shot, retraining-free method to compress monolingual code LLMs by factorizing large weight matrices into two smaller dense matrices. It shows that code LLMs (StarCoder and CodeGen) can sustain substantial rank reductions (up to ~39.6%) with minimal perplexity increase, enabling reductions from StarCoder 16B to 13.2B parameters without hurting HumanEval Pass@1 and achieving up to 22.35% inference speedups on A100 hardware. LoRD is demonstrated to be compatible with near-lossless quantization (SpQR) and with parameter-efficient fine-tuning methods like LoRA/QLoRA, offering additional memory and efficiency benefits. The work argues for LoRD as a practical, hardware-friendly compression paradigm for large code LLMs, with potential applicability beyond code-focused models and toward edge and green AI deployments.

Abstract

Low Rank Decomposition of matrix - splitting a large matrix into a product of two smaller matrix offers a means for compression that reduces the parameters of a model without sparsification, and hence delivering more speedup on modern hardware. Moreover, unlike quantization, the compressed linear layers remain fully differentiable and all the parameters trainable, while being able to leverage the existing highly efficient kernels over floating point matrices. We study the potential to compress Large Language Models (LLMs) for monolingual Code generation via Low Rank Decomposition (LoRD) and observe that ranks for the linear layers in these models can be reduced by upto 39.58% with less than 1% increase in perplexity. We then use Low Rank Decomposition (LoRD) to compress StarCoder 16B to 13.2B parameter with no drop and to 12.3B with minimal drop in HumanEval Pass@1 score, in less than 10 minutes on a single A100. The compressed models speeds up inference by up to 22.35% with just a single line of change in code over huggingface's implementation with pytorch backend. Low Rank Decomposition (LoRD) models remain compatible with state of the art near-lossless quantization method such as SpQR, which allows leveraging further compression gains of quantization. Lastly, QLoRA over Low Rank Decomposition (LoRD) model further reduces memory requirements by as much as 21.2% over vanilla QLoRA while offering similar gains from parameter efficient fine tuning. Our work shows Low Rank Decomposition (LoRD) as a promising new paradigm for LLM compression.

LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression

TL;DR

The paper proposes Low Rank Decomposition (LoRD) as a one-shot, retraining-free method to compress monolingual code LLMs by factorizing large weight matrices into two smaller dense matrices. It shows that code LLMs (StarCoder and CodeGen) can sustain substantial rank reductions (up to ~39.6%) with minimal perplexity increase, enabling reductions from StarCoder 16B to 13.2B parameters without hurting HumanEval Pass@1 and achieving up to 22.35% inference speedups on A100 hardware. LoRD is demonstrated to be compatible with near-lossless quantization (SpQR) and with parameter-efficient fine-tuning methods like LoRA/QLoRA, offering additional memory and efficiency benefits. The work argues for LoRD as a practical, hardware-friendly compression paradigm for large code LLMs, with potential applicability beyond code-focused models and toward edge and green AI deployments.

Abstract

Low Rank Decomposition of matrix - splitting a large matrix into a product of two smaller matrix offers a means for compression that reduces the parameters of a model without sparsification, and hence delivering more speedup on modern hardware. Moreover, unlike quantization, the compressed linear layers remain fully differentiable and all the parameters trainable, while being able to leverage the existing highly efficient kernels over floating point matrices. We study the potential to compress Large Language Models (LLMs) for monolingual Code generation via Low Rank Decomposition (LoRD) and observe that ranks for the linear layers in these models can be reduced by upto 39.58% with less than 1% increase in perplexity. We then use Low Rank Decomposition (LoRD) to compress StarCoder 16B to 13.2B parameter with no drop and to 12.3B with minimal drop in HumanEval Pass@1 score, in less than 10 minutes on a single A100. The compressed models speeds up inference by up to 22.35% with just a single line of change in code over huggingface's implementation with pytorch backend. Low Rank Decomposition (LoRD) models remain compatible with state of the art near-lossless quantization method such as SpQR, which allows leveraging further compression gains of quantization. Lastly, QLoRA over Low Rank Decomposition (LoRD) model further reduces memory requirements by as much as 21.2% over vanilla QLoRA while offering similar gains from parameter efficient fine tuning. Our work shows Low Rank Decomposition (LoRD) as a promising new paradigm for LLM compression.
Paper Structure (16 sections, 6 equations, 5 figures, 2 tables)

This paper contains 16 sections, 6 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Code LLMs are Low Rank Decomposable
  3. Background
  4. Experimental Settings
  5. Change in Perplexity across Reduction in Rank
  6. Compression and speedup through Decomposition
  7. Achieving compression and inference speedup
  8. Trends across different layers in a transformer block:
  9. Performance of compressed models
  10. Speedup from LoRD
  11. Combining LoRD with Quantization and LoRA
  12. Quantization
  13. Parameter Efficient tuning of LoRD models
  14. Related Work
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Perplexity vs %Reduction in Rank for Different Models.
  • Figure 2: Parity Point across various aspect ratios ($\alpha$) of the different linear layers in transformers.
  • Figure 3: Parameter Reduction vs perplexity for decomposition across various layers.
  • Figure 4: Time and Model size of StarCoder 16B across ranks.
  • Figure 5: LoRA vs LoRD + LoRA.