Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization

Advait Gadhikar, Riccardo Grazzi, James Hensman

TL;DR

This work proposes methods that learn fusible rotations by minimizing principled and cheap proxy objectives to the weight quantization error, and primarily focuses on GPTQ as the quantization method.

Abstract

The presence of outliers in Large Language Models (LLMs) weights and activations makes them difficult to quantize. Recent work has leveraged rotations to mitigate these outliers. In this work, we propose methods that learn fusible rotations by minimizing principled and cheap proxy objectives to the weight quantization error. We primarily focus on GPTQ as the quantization method. Our main method is OptRot, which reduces weight outliers simply by minimizing the element-wise fourth power of the rotated weights. We show that OptRot outperforms both Hadamard rotations and more expensive, data-dependent methods like SpinQuant and OSTQuant for weight quantization. It also improves activation quantization in the W4A8 setting. We also propose a data-dependent method, OptRot$^{+}$, that further improves performance by incorporating information on the activation covariance. In the W4A4 setting, we see that both OptRot and OptRot$^{+}$ perform worse, highlighting a trade-off between weight and activation quantization.

OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization

TL;DR

This work proposes methods that learn fusible rotations by minimizing principled and cheap proxy objectives to the weight quantization error, and primarily focuses on GPTQ as the quantization method.

Abstract

The presence of outliers in Large Language Models (LLMs) weights and activations makes them difficult to quantize. Recent work has leveraged rotations to mitigate these outliers. In this work, we propose methods that learn fusible rotations by minimizing principled and cheap proxy objectives to the weight quantization error. We primarily focus on GPTQ as the quantization method. Our main method is OptRot, which reduces weight outliers simply by minimizing the element-wise fourth power of the rotated weights. We show that OptRot outperforms both Hadamard rotations and more expensive, data-dependent methods like SpinQuant and OSTQuant for weight quantization. It also improves activation quantization in the W4A8 setting. We also propose a data-dependent method, OptRot, that further improves performance by incorporating information on the activation covariance. In the W4A4 setting, we see that both OptRot and OptRot perform worse, highlighting a trade-off between weight and activation quantization.
Paper Structure (25 sections, 10 theorems, 66 equations, 10 figures, 11 tables)

This paper contains 25 sections, 10 theorems, 66 equations, 10 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Quantization Error and Outliers
  4. Quantization Error Bounds
  5. OptRot
  6. Data-Free Objectives - OptRot
  7. Data-Dependent Objectives - OptRot$^{+}$
  8. Experiments
  9. Layerwise Results
  10. End-to-End Results
  11. Conclusion
  12. Round to Nearest Error Bound
  13. Approximating the KL Divergence with the Layerwise Errors
  14. Using Hessian Information for OptRot
  15. Difference Between True and Constrained LDL in Practice.
  16. ...and 10 more sections

Key Result

Theorem 3.1

Let $W \in \mathbb{R}^{m \times n}$ be a weight matrix and $H \in \mathbb{R}^{n \times n}$ be a symmetric positive semi-definite (PSD) matrix. If $s = w_{\mathrm{max}}$, then where $\mu_W$ is the weight incoherence and $\lambda_{\max}(H)$ is the maximum eigenvalue of the matrix $H$.

Figures (10)

  • Figure 1: Rotations applied by QuaRot, SpinQuant and OptRot to improve PTQ. $R_1, R_1^\top, R_2, R_2^\top$ and $R_4^\top$ (yellow) are fusible rotations. $R_3$ (red) and $R_4$ (orange) are online rotations. $R_1$ is shared across layers. In OptRot and SpinQuant $R_1$ and $R_2$ are learned without increasing inference cost.
  • Figure 2: Comparing the KL divergence of data-dependent objectives after learning rotations and $4$-bit weight quantization with GPTQ on Llama-3.2-1B.
  • Figure 3: Weight incoherence $\mu_W$ optimized by OptRot (top row), Hessian incoherence $\mu_H$ (second row), $\mathop{\mathrm{tr}}\nolimits(D) / \text{Tr}(H)$ (third row), the upper bound $\text{UB} / \text{Tr}(H)$ (fourth row) and the SNR after quantization with GPTQ (bottom row) for Llama-3.1-8B.
  • Figure 4: Comparing $\max_i e_i^\top L^\top Le_i$ for the true LDL versus the constrained LDL on a Llama-3.2-1B model.
  • Figure 5: Comparison among different bounds for $\mathop{\mathrm{tr}}\nolimits(D)$ when varying the dimension of $H$ ($n \times n$). Incoherence Bound is $\mu_H^2\mathop{\mathrm{tr}}\nolimits(H^{1/2})^2/n$ with $\mu_H = \max_{i,j}|Q_{i,j}| \sqrt{n}$ and $Q$ is an eigenvector matrix of $H$ which can be the one used to construct $H$ (V1) or the one output of the SVD scipy function (SVD). There is ambiguity whenever 2 or more eigenvalues coincide and the best choice is $Q$ that minimizes $\max_{i,j}|Q_{i,j}|$.
  • ...and 5 more figures

Theorems & Definitions (17)

  • Theorem 3.1: Worst Case Error Bound for RTN
  • Definition 3.2: Constrained LDL
  • Theorem 3.3: Theorem 14 in chee2023quip
  • Theorem 3.4
  • Corollary 3.5: GPTQ Bound
  • Theorem 1.1: Worst Case Error Bound for RTN
  • proof
  • Definition 3.1: Constrained LDL
  • Lemma 3.2: Lemma 12 from chee2023quip
  • proof
  • ...and 7 more