Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Model Compression via the Nested Activation-Aware Decomposition

Jun Lu, Tianyi Xu, Bill Ding, David Li, Yu Kang

TL;DR

This work tackles practical post-training compression of large language models by introducing a nested activation-aware decomposition (NSVD) for low-rank weight compression. It combines truncation-aware data whitening with a two-stage, shared-rank decomposition (ranks $k_1$ and $k_2$ with $k_1+k_2=k$) to reduce activation-induced bias, supported by activation-aware theory and Eckart-Young-Mirsky principles. NSVD variants (NSVD-I/NSVD-II) and flexible NID-I/NID-II options enable robust, training-free compression that matches or exceeds prior SVD-based methods (ASVD-0, ASVD-I) across eight datasets, six models, and three LLM families, particularly at 30–50% compression and in multilingual multitask settings. The results demonstrate significant perplexity reductions with comparable computational cost, highlighting the method’s practical value for deploying compressed LLMs in diverse languages and tasks.

Abstract

In this paper, we tackle the critical challenge of compressing large language models (LLMs) to facilitate their practical deployment and broader adoption. We introduce a novel post-training compression paradigm that focuses on low-rank decomposition of LLM weights. Our analysis identifies two main challenges in this task: the variability in LLM activation distributions and handling unseen activations from different datasets and models. To address these challenges, we propose a nested activation-aware framework (NSVD) for LLMs, a training-free approach designed to enhance the accuracy of low-rank decompositions by managing activation outliers through transforming the weight matrix based on activation distribution and the original weight matrix. This method allows for the absorption of outliers into the transformed weight matrix, improving decomposition accuracy. Our comprehensive evaluation across eight datasets and six models from three distinct LLM families demonstrates the superiority of NSVD over current state-of-the-art methods, especially at medium to large compression ratios or in multilingual and multitask settings.

Large Language Model Compression via the Nested Activation-Aware Decomposition

TL;DR

This work tackles practical post-training compression of large language models by introducing a nested activation-aware decomposition (NSVD) for low-rank weight compression. It combines truncation-aware data whitening with a two-stage, shared-rank decomposition (ranks and with ) to reduce activation-induced bias, supported by activation-aware theory and Eckart-Young-Mirsky principles. NSVD variants (NSVD-I/NSVD-II) and flexible NID-I/NID-II options enable robust, training-free compression that matches or exceeds prior SVD-based methods (ASVD-0, ASVD-I) across eight datasets, six models, and three LLM families, particularly at 30–50% compression and in multilingual multitask settings. The results demonstrate significant perplexity reductions with comparable computational cost, highlighting the method’s practical value for deploying compressed LLMs in diverse languages and tasks.

Abstract

In this paper, we tackle the critical challenge of compressing large language models (LLMs) to facilitate their practical deployment and broader adoption. We introduce a novel post-training compression paradigm that focuses on low-rank decomposition of LLM weights. Our analysis identifies two main challenges in this task: the variability in LLM activation distributions and handling unseen activations from different datasets and models. To address these challenges, we propose a nested activation-aware framework (NSVD) for LLMs, a training-free approach designed to enhance the accuracy of low-rank decompositions by managing activation outliers through transforming the weight matrix based on activation distribution and the original weight matrix. This method allows for the absorption of outliers into the transformed weight matrix, improving decomposition accuracy. Our comprehensive evaluation across eight datasets and six models from three distinct LLM families demonstrates the superiority of NSVD over current state-of-the-art methods, especially at medium to large compression ratios or in multilingual and multitask settings.

Paper Structure

This paper contains 11 sections, 4 theorems, 10 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. (2).
  5. Experiments
  6. Performance under different compression ratios
  7. Performance under different values of $k_1$
  8. Performance under different values of $k_1$ for NID
  9. Performance on different LLMs
  10. Performance on different scales of LLMs
  11. Conclusion

Key Result

Theorem 1

Given a matrix $\bm{A}\in \mathbb{R}^{m\times n}$, $1\leq k\leq \mathrm{rank}(\bm{A})=r$, and let $\bm{A}_k$ be the truncated SVD (TSVD) of $\bm{A}$ retaining the largest $k$ singular terms, i.e., $\bm{A}_k = \sum_{i=1}^{k} \sigma_i\bm{u}_i\bm{v}_i^\top$ from the SVD of $\bm{A}=\sum_{i=1}^{r} \sigma

Figures (1)

  • Figure 1: The cosine similarity between the activations from the calibration dataset (the training set of WikiText-2) and those from various evaluation datasets. Table \ref{['tab:nsce_ppl0']} presents the mean of similarities and the standard deviations. Specifically, the calibration set shows an average similarity of less than 0.5 with both CMRC (CN) and AlpacaEval (JP).

Theorems & Definitions (4)

  • Theorem 1: Eckart-Young-Mirsky Theorem
  • Theorem 2: ASVD-I: Activation-Aware Low-Rank Approximation by Cholesky
  • Theorem 3: ASVD-II: Activation-Aware Low-Rank Approximation by SVD
  • Theorem 4: ASVD-III: Activation-Aware Low-Rank Approximation via Scaling Eigenvalues