Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantization of Large Language Models with an Overdetermined Basis

Daniil Merkulov, Daria Cherniuk, Alexander Rudikov, Ivan Oseledets, Ekaterina Muravleva, Aleksandr Mikhalev, Boris Kashin

TL;DR

The paper tackles the memory and compute bottlenecks of large language models by introducing Kashin Quantization, a data-quantization method that decomposes inputs into two factors via $x \approx u + Qv$ with tightly bounded infinity norms. It extends to matrices with a Matrix Decomposition Kashin Algorithm that uses a Kronecker-factorized orthogonal transform $Q = Q_2^T \otimes Q_1^T$ to reduce memory and accelerate matvecs, enabling centroid-based quantization of the resulting factors. The approach is empirically validated on OPT models and GLUE tasks, showing competitive perplexity and accuracy at low bitrates, though convergence can vary across layers depending on the chosen orthogonal basis. The work suggests Kashin Quantization as a promising direction for efficient LLM quantization and potential activation quantization, with implications for on-device inference and federated/distributed settings.

Abstract

In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids for quantization purposes. We study the theoretical properties of the proposed approach and rigorously evaluate our compression algorithm in the context of next-word prediction tasks and on a set of downstream tasks for text classification. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance while ensuring data compression, marking a significant advancement in the field of data quantization.

Quantization of Large Language Models with an Overdetermined Basis

TL;DR

The paper tackles the memory and compute bottlenecks of large language models by introducing Kashin Quantization, a data-quantization method that decomposes inputs into two factors via with tightly bounded infinity norms. It extends to matrices with a Matrix Decomposition Kashin Algorithm that uses a Kronecker-factorized orthogonal transform to reduce memory and accelerate matvecs, enabling centroid-based quantization of the resulting factors. The approach is empirically validated on OPT models and GLUE tasks, showing competitive perplexity and accuracy at low bitrates, though convergence can vary across layers depending on the chosen orthogonal basis. The work suggests Kashin Quantization as a promising direction for efficient LLM quantization and potential activation quantization, with implications for on-device inference and federated/distributed settings.

Abstract

In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or tensor into two factors. The first factor maintains a small infinity norm, while the second exhibits a similarly constrained norm when multiplied by an orthogonal matrix. Surprisingly, the entries of factors after decomposition are well-concentrated around several peaks, which allows us to efficiently replace them with corresponding centroids for quantization purposes. We study the theoretical properties of the proposed approach and rigorously evaluate our compression algorithm in the context of next-word prediction tasks and on a set of downstream tasks for text classification. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance while ensuring data compression, marking a significant advancement in the field of data quantization.
Paper Structure (16 sections, 1 theorem, 15 equations, 8 figures, 3 tables, 2 algorithms)

This paper contains 16 sections, 1 theorem, 15 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Representing the vector in overdetermined basis.
  4. Computing the representation using a greedy approach.
  5. Convergence of the algorithm
  6. Matrix version of the algorithm.
  7. How orthogonal matrix $Q$ affects the algorithm
  8. Random Orthogonal Matrix
  9. Householder Reflection
  10. Discrete Cosine Transform
  11. Butterfly matrices
  12. Convergence properties
  13. Quantization approach
  14. Experiments
  15. Related Work
  16. ...and 1 more sections

Key Result

Theorem 1

For $\forall x \in B^N_2 = \{x \in \mathbb{R}^N : \Vert x\Vert_2 \le 1\}$ a greedy algorithm in $k$ steps builds vectors $u_k$ and $v_k$ such that where $c > 0$ is an absolute constant, $Q \in O^N$, where $O^N$ is a set of orthogonal matrices in $\mathbb{R}^N$ with Haar measure $\mu_H$.

Figures (8)

  • Figure 1: We select the vector to project on in a greedy manner.
  • Figure 2: Convergence criterion ($\Vert x_0 - u_k - \hat{v}_k\Vert _2$) for randomly generated vector $x$ of length $1000$. The semi-transparent area shows the confidence interval.
  • Figure 3: Convergence criterion ($\Vert x - \hat{x}\Vert_2$) of the matrix decomposition Kashin algorithm for different layers of OPT-125m model. Left: DCT matrix converges very slowly after several iterations. Right: DCT matrix converges well for another layer.
  • Figure 4: Left: Matrix Decomposition Kashin Algorithm \ref{['alg:kashin_mat']} performs significantly faster for matrices, than vectorization and applying Vector Decomposition Kashin Algorithm \ref{['alg:kashin_vec']}. Right: Time comparison of different orthogonal matrices for Matrix Decomposition Kashin Algorithm
  • Figure 5: Butterfly factor matrices from left to right: $B_8$, $B_4$, $B_2$
  • ...and 3 more figures

Theorems & Definitions (1)

  • Theorem 1: kashin2023efficient