Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

Shih-Yang Liu, Maksim Khadkevich, Nai Chit Fung, Charbel Sakr, Chao-Han Huck Yang, Chien-Yi Wang, Saurav Muralidharan, Hongxu Yin, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, Pavlo Molchanov, Min-Hung Chen

TL;DR

EoRA is proposed, a novel fine-tuning-free method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats.

Abstract

While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA, a novel fine-tuning-free method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., $\mathbf{10.84\%}$ on ARC-Challenge, $\mathbf{6.74\%}$ on MathQA, and $\mathbf{11.45\%}$ on GSM8K) for LLaMA3-8B compressed to 3-bit. We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA.

EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

TL;DR

EoRA is proposed, a novel fine-tuning-free method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats.

Abstract

While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA, a novel fine-tuning-free method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., on ARC-Challenge, on MathQA, and on GSM8K) for LLaMA3-8B compressed to 3-bit. We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA.

Paper Structure

This paper contains 34 sections, 9 equations, 3 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries: Post-training Compression
  3. Method: EoRA
  4. Mapping EoRA loss (Eq. \ref{['eq:overall_eq']}) to task-specific compression loss (Eq. \ref{['eq:layer_objective']}):
  5. Experiments
  6. Experiments Details
  7. Main Results
  8. Sparsity Error Compensation
  9. Quantization Error Compensation
  10. Ablation Study: Ranks and Calibration Sizes
  11. EoRA as LoRA initialization for Fine-tuning Compressed Models
  12. Kernel Optimization, Inference Speed Evaluation and Memory Overhead of EoRA
  13. Related Works
  14. Post-training LLM Compression:
  15. Low-Rank Decomposition:
  16. ...and 19 more sections

Figures (3)

  • Figure 1: An overview of our proposed EoRA, which enables swift task-specific accuracy enhancement for compressed LLMs without fine-tuning, using only a small amount of downstream calibration data. At inference time, a single compressed backbone is loaded, while lightweight, task-specific low-rank modules can be dynamically toggled on and off on demand, enabling efficient and flexible deployment. EoRA with rank 128 boosts the accuracy of the LLaMA3-8B model pruned to $2{:}4$ structured sparsity by $4.53\%$, $3.48\%$, and $11.83\%$ on ARC-C, MathQA, and GSM8K, respectively—all achieved within minutes using just 64 calibration samples per task.
  • Figure 2: Results of applying EoRA and other baselines with rank set to {64,128,256,512} to improve LLaMA3-8B models pruned to 2:4 sparsity by SparseGPT on (a) ARC-C/(b) MathQA/(c) GSM8K.
  • Figure 3: (a) We propose fusing the multiplication of $B$ with the weight quantization kernel to minimize data movement overhead and substantially improve the inference latency. (b) The model size and ARC-C accuracy of EoRA with rank 128/512, quantized to 4-bit for compensating LLaMA3-8B quantized to 4/3-bit or pruned to 2:4 sparsity.