High-Rate Quantized Matrix Multiplication: Theory and Practice

TL;DR

This work establishes information-theoretic benchmarks for quantized matrix multiplication (MatMul) in two practical settings relevant to LLMs: generic MatMul requiring joint weight+activation quantization and weight-only quantization using the covariance Σ_X of activations. It derives fundamental rate–distortion limits, compares a range of practical schemes (INT/FP, NV, NestQuant), and introduces WaterSIC to achieve near-optimal WMSE performance with a Σ_X-aware, per-coordinate scheduling strategy. In the high-rate regime, distortion scales as D^* ≈ |Σ_X|^{1/n} σ_W^2 2^{-2R}, and WaterSIC attains within a small (≈0.25 bit) gap of this limit, even under universal decoding constraints. The paper also provides extensive numerical validation on Llama-3-8B calibration data, clarifying when rotations help or hurt different quantizers and offering concrete guidance for deploying near-optimal quantized MatMul in large-scale models.

Abstract

This work investigates the problem of quantized matrix multiplication (MatMul), which has become crucial for the efficient deployment of large language models (LLMs). We consider two settings: 1) Generic MatMul, where both matrices must be quantized (weight+activation quantization); and 2) weight-only quantization, where the second matrix is only known through covariance matrix $Σ_X$ of its columns. For each setting, we first review the fundamental information-theoretic tradeoff between quantization rate and distortion (high-rate theory), and then analyze the performance of several popular quantization schemes, comparing them to these fundamental limits. Specifically, we discuss rate loss (compared to information theoretic optima) of absmax INT and floating-point (FP) quantization, for which we also derive remarkably accurate heuristic approximations. Weight-only quantization is related to the problem of weighted mean squared error (WMSE) source coding, whose classical (reverse) waterfilling solution dictates how one should distribute rate between coordinates of the vector. We show how waterfilling can be used to improve practical LLM quantization algorithms (GPTQ), which at present allocate rate equally. This new scheme (termed ``WaterSIC'') only uses scalar INT quantizers, but its high-rate performance is basis free (it depends only on the determinant of $Σ_X$ and, thus, unlike existing schemes, is immune to applying random rotations) and is within a multiplicative factor of $\frac{2πe}{12}$ (or 0.25 bit/entry) of the information-theoretic distortion limit (!). GPTQ's performance is affected by the choice of basis, but for a random rotation and actual $Σ_X$ from Llama-3-8B we find GPTQ to be within 0.1 bit (depending on the layer type) of WaterSIC, suggesting that GPTQ with random rotation is also near optimal (for high-rate quantization).

Figures (10)

• Figure 1: Distribution of $\epsilon$-quantization errors of a vector $\mathcal{N}(0,\sigma^2 I_{10000})$ with $\sigma^2 = 1/12$ and $\epsilon=0.1$ and effects of shrinkage on MSE.
• Figure 2: Demonstrating accuracy of FP8 approximation \ref{['eq:DfpMGauss']}. The figure plots average ratio (over 1024 pairs of vectors of dimension 32, each generated iid $\mathcal{N}(0,\sigma^2)$) of the normalized squared error of the inner-product (see left-hand side of \ref{['eq:DfpMGauss']}) against the simple theoretical approximation $2^{-2R_\mathrm{eff}(\mathrm{FP},\mathcal{M})) + 1}$.
• Figure 3: Histograms of $\Delta_{\mathrm{FP}}$ and $\Delta_{\mathrm{INT}}$ for the setup described in Section \ref{['subsec:numeric']}.
• Figure 4: Performance of several quantization schemes with $R=4.5$.
• Figure 5: Illustrating $\Sigma_X$ of activations entering various layers of Llama-3-8B when processing Wikitext-2 dataset. Note that this is an estimate of the rate advantage assumes weight matrices are well modeled by $\mathcal{N}(0,I_n)$. In particular, actual weight matrices were never used for this plot.

Theorems & Definitions (3)

• Proposition 1: Zador
• Lemma 1
• Lemma 2