Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization

Shihao Zhang, Haoyu Zhang, Ian Colbert, Rayan Saab

TL;DR

It is proved that Qronos admits an efficient implementation that uses the Cholesky decomposition for solving least-squares problems, and it is demonstrated that Qronos is compatible with existing transformation techniques such as Hadamard-based incoherence processing and weight-activation scaling equalization, among others.

Abstract

We introduce Qronos -- a new state-of-the-art post-training quantization algorithm that sequentially rounds and updates neural network weights. Qronos not only explicitly corrects errors due to both weight and activation quantization, but also errors resulting from quantizing previous layers. Our iterative algorithm is based on an interpretable and disciplined optimization framework that subsumes and surpasses existing data-driven approaches. At each step, Qronos alternates between error correction and diffusion via optimal update rules. Importantly, we prove that Qronos admits an efficient implementation that uses the Cholesky decomposition for solving least-squares problems. We also demonstrate that Qronos is compatible with existing transformation techniques such as Hadamard-based incoherence processing and weight-activation scaling equalization, among others. We evaluate Qronos using recent autoregressive language generation models in the Llama3 family; Qronos consistently outperforms previous state-of-the-art adaptive rounding methods when quantizing the weights, activations, and/or KV caches.

Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization

TL;DR

It is proved that Qronos admits an efficient implementation that uses the Cholesky decomposition for solving least-squares problems, and it is demonstrated that Qronos is compatible with existing transformation techniques such as Hadamard-based incoherence processing and weight-activation scaling equalization, among others.

Abstract

We introduce Qronos -- a new state-of-the-art post-training quantization algorithm that sequentially rounds and updates neural network weights. Qronos not only explicitly corrects errors due to both weight and activation quantization, but also errors resulting from quantizing previous layers. Our iterative algorithm is based on an interpretable and disciplined optimization framework that subsumes and surpasses existing data-driven approaches. At each step, Qronos alternates between error correction and diffusion via optimal update rules. Importantly, we prove that Qronos admits an efficient implementation that uses the Cholesky decomposition for solving least-squares problems. We also demonstrate that Qronos is compatible with existing transformation techniques such as Hadamard-based incoherence processing and weight-activation scaling equalization, among others. We evaluate Qronos using recent autoregressive language generation models in the Llama3 family; Qronos consistently outperforms previous state-of-the-art adaptive rounding methods when quantizing the weights, activations, and/or KV caches.
Paper Structure (19 sections, 8 theorems, 49 equations, 3 figures, 11 tables, 2 algorithms)

This paper contains 19 sections, 8 theorems, 49 equations, 3 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Qronos
  4. Algorithm and Efficient Implementation
  5. Theoretical Interpretation and Intuition
  6. Experiments
  7. Weight-Only Quantization
  8. Weight-Activation Quantization
  9. Hardware Efficiency and Runtime Analysis
  10. Conclusions
  11. Pseudocode of Qronos
  12. Results on Additional Models
  13. Experiment Details for Quantization Transforms
  14. Grid scaling ablation study for 2 bits and fewer
  15. More on Why Qronos Outperforms OPTQ
  16. ...and 4 more sections

Key Result

Theorem 3.1

Let $(q_t, w^{(t-1)}_{\geq t})$ be the iterates generated by Equation original_bidq_update_q and Equation original_bidq_update_w, with initialization $w^{(0)}_{\geq 1} = w$. Define an alternative sequence $(\hat{q}_t, \hat{w}^{(t-1)}_{\geq t})$ using the same initialization $\hat{w}^{(0)}_{\geq 1} = and, for $t = 2, \dots, N$, define Then for $t = 1, \dots, N$, the two procedures yield identical

Figures (3)

  • Figure 1: The modern quantization pipeline is typically a two-stage process consisting of (1) transformations that make weights and/or activations more amenable to quantization, followed by (2) rounding functions that map weights and/or activations onto a quantization grid.
  • Figure 2: We compare the runtime of (a) the rounding algorithm and (b) the overall quantization pipeline as we scale the input features $N$, as measured on an AMD MI210. We average all measurements over 3 seeds and normalize to the runtime of OPTQ where $N=32$.
  • Figure 3: We visualize the evolution of the average relative error over transformer blocks when quantizing the Llama3 1B foundation model to 3 bits, further discussed in \ref{['appendix:optq_vs_qronos']}.

Theorems & Definitions (17)

  • Theorem 3.1
  • Lemma 3.2: Equivalence of Least-Squares Formulation and Cholesky Formulation
  • Remark 3.3: Memory Efficiency
  • Corollary 3.4
  • Proposition E.1
  • proof
  • Proposition E.2
  • proof
  • proof
  • Lemma G.1
  • ...and 7 more