Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeUQI: Near-Optimal Uniform Quantization Parameter Initialization

Li Lin, Xinyu Hu, Xiaojun Wan

TL;DR

Post-training uniform quantization enables memory- and latency-efficient deployment of large language models, but initialization—particularly the integer constraint on the zero-point—limits performance. NeUQI reframes initialization as a two-stage optimization over $z$ and $s$ using a diagonal Hessian approximation, removing the integer constraint and enabling near-optimal quantization parameters that integrate with existing PTQ pipelines. Empirically, NeUQI delivers consistent improvements across LLaMA 2/3 and Qwen 2.5 at 2-bit and 3-bit settings, and, when combined with lightweight distillation, can outperform PV-tuning with far lower resource demands. This work facilitates practical deployment of ultra-efficient LLMs on consumer hardware by improving quantization quality without heavy retraining.

Abstract

Large language models (LLMs) achieve impressive performance across domains but face significant challenges when deployed on consumer-grade GPUs or personal devices such as laptops, due to high memory consumption and inference costs. Post-training quantization (PTQ) of LLMs offers a promising solution that reduces their memory footprint and decoding latency. In practice, PTQ with uniform quantization representation is favored for its efficiency and ease of deployment since uniform quantization is widely supported by mainstream hardware and software libraries. Recent studies on $\geq 2$-bit uniform quantization have led to noticeable improvements in post-quantization model performance; however, they primarily focus on quantization methodologies, while the initialization of quantization parameters is underexplored and still relies on the suboptimal Min-Max strategies. In this work, we propose NeUQI, a method devoted to efficiently determining near-optimal initial parameters for uniform quantization. NeUQI is orthogonal to prior quantization methodologies and can seamlessly integrate with them. The experiments with the LLaMA and Qwen families on various tasks demonstrate that our NeUQI consistently outperforms existing methods. Furthermore, when combined with a lightweight distillation strategy, NeUQI can achieve superior performance to PV-tuning, a much more resource-intensive approach.

NeUQI: Near-Optimal Uniform Quantization Parameter Initialization

TL;DR

Post-training uniform quantization enables memory- and latency-efficient deployment of large language models, but initialization—particularly the integer constraint on the zero-point—limits performance. NeUQI reframes initialization as a two-stage optimization over and using a diagonal Hessian approximation, removing the integer constraint and enabling near-optimal quantization parameters that integrate with existing PTQ pipelines. Empirically, NeUQI delivers consistent improvements across LLaMA 2/3 and Qwen 2.5 at 2-bit and 3-bit settings, and, when combined with lightweight distillation, can outperform PV-tuning with far lower resource demands. This work facilitates practical deployment of ultra-efficient LLMs on consumer hardware by improving quantization quality without heavy retraining.

Abstract

Large language models (LLMs) achieve impressive performance across domains but face significant challenges when deployed on consumer-grade GPUs or personal devices such as laptops, due to high memory consumption and inference costs. Post-training quantization (PTQ) of LLMs offers a promising solution that reduces their memory footprint and decoding latency. In practice, PTQ with uniform quantization representation is favored for its efficiency and ease of deployment since uniform quantization is widely supported by mainstream hardware and software libraries. Recent studies on -bit uniform quantization have led to noticeable improvements in post-quantization model performance; however, they primarily focus on quantization methodologies, while the initialization of quantization parameters is underexplored and still relies on the suboptimal Min-Max strategies. In this work, we propose NeUQI, a method devoted to efficiently determining near-optimal initial parameters for uniform quantization. NeUQI is orthogonal to prior quantization methodologies and can seamlessly integrate with them. The experiments with the LLaMA and Qwen families on various tasks demonstrate that our NeUQI consistently outperforms existing methods. Furthermore, when combined with a lightweight distillation strategy, NeUQI can achieve superior performance to PV-tuning, a much more resource-intensive approach.

Paper Structure

This paper contains 42 sections, 1 theorem, 41 equations, 2 figures, 10 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Uniform Quantization
  4. Quantization Parameter Initialization
  5. Background
  6. Uniform Quantization
  7. Min-Max Initialization
  8. Method
  9. Quantization Loss
  10. NeUQI
  11. Optimization of Zero-Point
  12. Optimization of Scale
  13. Lightweight Distillation
  14. Experiment
  15. Baselines and Evaluation
  16. ...and 27 more sections

Key Result

Lemma 1

If weights are independently and uniformly drawn from an interval $[a, b]$, then the optimal parameters to for $k$-bit uniform quantization to minimize excepted quantization error $\mathbb{E}_{x\sim \mathcal{U}(a,b)}[(Q_{s,z}(x)-x)^2]$ are given by

Figures (2)

  • Figure 1: Perplexity on C4 (left) and average accuracy across five common benchmarks (right) with Qwen 2.5 family, plotted against the base-2 logarithm ($\log_2$) of model memory usage on the x-axis. The results include the non-quantized models (BF16) and the quantized models using commonly-used GPTQ, or our proposed NeUQI (both 3-bit). More details and results are shown in Section \ref{['sec:experiment']}.
  • Figure 2: Line plots of $\mathcal{L}(s, z^*(s))$ versus $s$ for the query projection matrix in the 15th transformer block. The horizontal axis shows the ratio of $s$ to the baseline value $(\text{max}(x) - \text{min}(x))/(2^k - 1)$. The left, center, and right plots correspond to the 1st, 1203rd, and 3709th rows of the query projection matrix. Local minima are indicated with red star (★).

Theorems & Definitions (1)

  • Lemma 1