Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ResSVD: Residual Compensated SVD for Large Language Model Compression

Haolei Bai, Siyong Jian, Tuo Liang, Yu Yin, Huan Wang

TL;DR

The paper tackles the challenge of compressing large language models with minimal performance loss by introducing ERC-SVD, a post-training SVD-based method. It pairs residual compensation for SVD truncation with partial-layer compression, selectively compressing the last few layers to curb error propagation. The approach achieves consistent improvements over prior SVD-based baselines across multiple LLM families and benchmarks, and remains compatible with quantization while delivering inference speedups. These results support practical deployment of compressed LLMs in resource-constrained settings without retraining.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in a wide range of downstream natural language processing tasks. Nevertheless, their considerable sizes and memory demands hinder practical deployment, underscoring the importance of developing efficient compression strategies. Singular value decomposition (SVD) decomposes a matrix into orthogonal components, enabling efficient low-rank approximation. This is particularly suitable for LLM compression, where weight matrices often exhibit significant redundancy. However, current SVD-based methods neglect the residual matrix from truncation, resulting in significant truncation loss. Additionally, compressing all layers of the model results in severe performance degradation. To overcome these limitations, we propose ResSVD, a new post-training SVD-based LLM compression method. Specifically, we leverage the residual matrix generated during the truncation process to reduce truncation loss. Moreover, under a fixed overall compression ratio, we selectively compress the last few layers of the model, which mitigates error propagation and significantly improves the performance of compressed models. Comprehensive evaluations of ResSVD on diverse LLM families and multiple benchmark datasets indicate that ResSVD consistently achieves superior performance over existing counterpart methods, demonstrating its practical effectiveness.

ResSVD: Residual Compensated SVD for Large Language Model Compression

TL;DR

The paper tackles the challenge of compressing large language models with minimal performance loss by introducing ERC-SVD, a post-training SVD-based method. It pairs residual compensation for SVD truncation with partial-layer compression, selectively compressing the last few layers to curb error propagation. The approach achieves consistent improvements over prior SVD-based baselines across multiple LLM families and benchmarks, and remains compatible with quantization while delivering inference speedups. These results support practical deployment of compressed LLMs in resource-constrained settings without retraining.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in a wide range of downstream natural language processing tasks. Nevertheless, their considerable sizes and memory demands hinder practical deployment, underscoring the importance of developing efficient compression strategies. Singular value decomposition (SVD) decomposes a matrix into orthogonal components, enabling efficient low-rank approximation. This is particularly suitable for LLM compression, where weight matrices often exhibit significant redundancy. However, current SVD-based methods neglect the residual matrix from truncation, resulting in significant truncation loss. Additionally, compressing all layers of the model results in severe performance degradation. To overcome these limitations, we propose ResSVD, a new post-training SVD-based LLM compression method. Specifically, we leverage the residual matrix generated during the truncation process to reduce truncation loss. Moreover, under a fixed overall compression ratio, we selectively compress the last few layers of the model, which mitigates error propagation and significantly improves the performance of compressed models. Comprehensive evaluations of ResSVD on diverse LLM families and multiple benchmark datasets indicate that ResSVD consistently achieves superior performance over existing counterpart methods, demonstrating its practical effectiveness.

Paper Structure

This paper contains 38 sections, 2 theorems, 16 equations, 10 figures, 15 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Techniques for Large Language Model Compression
  4. SVD-based Techniques for Compressing LLMs
  5. ERC-SVD
  6. Residual Compensation for SVD Truncation
  7. Preliminaries for SVD.
  8. Explanation for Residual Compensation.
  9. Partial-layer Compression for SVD
  10. Experiments
  11. Setups
  12. Evaluation Details.
  13. Implementation Details.
  14. Results
  15. Performance under different compression ratios.
  16. ...and 23 more sections

Key Result

Lemma 1

Eckart-Young-Mirsky Theoremhorn2012matrix. Let $\boldsymbol{A}\in\mathbb{R}^{m\times n}$ be a matrix and $r\le\min(m,n)$ be a given rank. If $\boldsymbol{A}_r$ denotes the optimal rank-$r$ approximation of $\boldsymbol{A}$ by SVD, then for any matrix $\boldsymbol{B}$ of rank $r$, the following inequ

Figures (10)

  • Figure 2: The overall framework of ERC-SVD, and comparison with other methods. The last $k$ layers are selected through partial-layer compression and compressed using residual compensation with calibration data. denotes these layers remain intact, while denotes these layers are replaced by low-rank approximations. The overall compression ratio is $R_o$, for ERC-SVD, the first $(N-k)$ layers stay unchanged, and the layer compression ratio $R_l$ for last $k$ layers is $(N \cdot R_o)/k$.
  • Figure 2: Perplexity ($\downarrow$) of different LLM structures under 30% compression ratio.
  • Figure 3: Layer-wise error comparison between the original model, LLaMA-7B, and OPT-6.7B compressed by ERC-SVD with different layer selection strategies on WikiText-2. The overall compression ratio is 20%, and all layer selection strategies strictly adhere to the compression constraint.
  • Figure 4: Layer-wise error comparison between the original model, Mistral-7B, and Vicuna-7B compressed by ERC-SVD with different layer selection strategies on WikiText-2. The overall compression ratio is 20%, and all layer selection strategies strictly adhere to the compression constraint.
  • Figure 5: Throughput of LLaMA-7B and its 40% compressed versions. The sequence length is 32.
  • ...and 5 more figures

Theorems & Definitions (3)

  • Lemma 1
  • Lemma 2
  • proof