Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Sparse Plus Low Rank Compression of LLM

Pawan Kumar, Aditi Gupta

TL;DR

To address memory and compute bottlenecks in large language models, this paper introduces Hierarchical Sparse Plus Low Rank (sHSS) compression, which first masks the largest-magnitude weights and then applies a recursive Hierarchically Semi-Separable (HSS) factorization to the residual. It proposes a three-level HSS construction and a Reverse Cuthill-McKee (RCM) reordering to maximize off-diagonal compressibility, enabling end-to-end training with standard optimizers on fp16. Experiments on LLaMA-7B focusing on self-attention projections demonstrate substantial storage reductions with perplexity comparable to the uncompressed model and improvements over dense and sparse-SVD baselines, including a representative perplexity of 1.64 at 30% sparsity and rank 512. The approach is hardware-friendly, supports fast matrix-vector products, and offers a scalable path toward deploying and fine-tuning larger foundation models on commodity hardware, potentially in combination with quantization and dynamic rank adaptation.

Abstract

Modern large language models (LLMs) place extraordinary pressure on memory and compute budgets, making principled compression indispensable for both deployment and continued training. We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that (i) removes the largest-magnitude weights into a sparse matrix S and (ii) applies a recursive Hierarchically Sparse Separable (HSS) low-rank factorisation to the dense residual matrix. A recursive rank-reducing strategy and a reverse Cuthill-Mckee (RCM) permutation are introduced to align high weights towards the diagonal with the block-diagonal hierarchy, maximising off-diagonal compressibility (because they are touched only once). HSS is hardware-friendly: its matrix-vector multiply reduces to one sparse and a sequence of thin-matrix multiplications and can be trained end-to-end with standard optimisers. Experiments on LLaMA-7B show that targeting only the self-attention projections (1.6 B parameters of Q, K, and V matrices out of a total 7B parameters) suffices to yield large memory savings while retaining comparable state-of-the-art perplexity scores on test samples of the WikiText dataset. For example, with a 30\% sparsity budget and an outer rank of 512, sHSS-RCM achieves a perplexity of 1.64, outperforming dense baselines and classical sparse-plus-SVD variants, while also achieving significant memory savings.

Hierarchical Sparse Plus Low Rank Compression of LLM

TL;DR

To address memory and compute bottlenecks in large language models, this paper introduces Hierarchical Sparse Plus Low Rank (sHSS) compression, which first masks the largest-magnitude weights and then applies a recursive Hierarchically Semi-Separable (HSS) factorization to the residual. It proposes a three-level HSS construction and a Reverse Cuthill-McKee (RCM) reordering to maximize off-diagonal compressibility, enabling end-to-end training with standard optimizers on fp16. Experiments on LLaMA-7B focusing on self-attention projections demonstrate substantial storage reductions with perplexity comparable to the uncompressed model and improvements over dense and sparse-SVD baselines, including a representative perplexity of 1.64 at 30% sparsity and rank 512. The approach is hardware-friendly, supports fast matrix-vector products, and offers a scalable path toward deploying and fine-tuning larger foundation models on commodity hardware, potentially in combination with quantization and dynamic rank adaptation.

Abstract

Modern large language models (LLMs) place extraordinary pressure on memory and compute budgets, making principled compression indispensable for both deployment and continued training. We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that (i) removes the largest-magnitude weights into a sparse matrix S and (ii) applies a recursive Hierarchically Sparse Separable (HSS) low-rank factorisation to the dense residual matrix. A recursive rank-reducing strategy and a reverse Cuthill-Mckee (RCM) permutation are introduced to align high weights towards the diagonal with the block-diagonal hierarchy, maximising off-diagonal compressibility (because they are touched only once). HSS is hardware-friendly: its matrix-vector multiply reduces to one sparse and a sequence of thin-matrix multiplications and can be trained end-to-end with standard optimisers. Experiments on LLaMA-7B show that targeting only the self-attention projections (1.6 B parameters of Q, K, and V matrices out of a total 7B parameters) suffices to yield large memory savings while retaining comparable state-of-the-art perplexity scores on test samples of the WikiText dataset. For example, with a 30\% sparsity budget and an outer rank of 512, sHSS-RCM achieves a perplexity of 1.64, outperforming dense baselines and classical sparse-plus-SVD variants, while also achieving significant memory savings.
Paper Structure (14 sections, 8 equations, 3 figures)

This paper contains 14 sections, 8 equations, 3 figures.

Table of Contents

  1. Introduction and Related Work
  2. Motivation for Hierarchical Low Rank
  3. Low Rank Compression Methods
  4. Introduction to HSS and sparse-HSS
  5. Introduction
  6. Background: Two-Level HSS
  7. Three-Level HSS Construction
  8. HSS Matrix-Vector Multiplication
  9. Sparse plus Hierarchical Low Rank
  10. Numerical Experiments
  11. Evaluation Dataset
  12. Storage versus Accuracy
  13. Role of Sparsity and Rank on PPL Scores
  14. Role of RCM Reordering

Figures (3)

  • Figure 1: Plot of low-rankness in off-diagonal blocks of attention matrix for German-English translation. We can see a weaker correlation of attention to far away tokens (see as lesser attention scores in the attention matrix). Not to say that all far away tokens are not relevant, but often only a few are relevant, hence, indicating an opportunity for possible low rank compression of off-diagonal blocks. This figure was generated using the publicly available Google Colab notebook code gala2021attention.
  • Figure 2: Ablation study for rank 512, depth = 4 for Hierarchical methods sHSS, sHSS-RCM. And sp10, sp20, sp30 show the percentage of the largest entries of weight stored in the separate matrix $S.$
  • Figure 3: Storage vs accuracy plot. In this plot, we observe that in general, hierarchical methods like sHSS and RCM ordered methods like sHSS-RCM have better perplexity for a given similar range of storage requirements compared to sparse plus low rank using SVD, and for the randomised variant sR-SVD. Note that it is not possible to have comparisons with the same storage requirements. We also observe that often RCM reordering, denoted by the purple shaded diamond, has lower perplexity scores compared to the one without RCM, the green shaded square boxes.