Two-Stage Grid Optimization for Group-wise Quantization of LLMs

TL;DR

This work addresses suboptimal group-wise quantization in GPTQ caused by ignoring input statistics and cross-group correlations. It proposes a two-stage approach: Stage 1 initializes each group scale to minimize local, input-aware reconstruction losses, and Stage 2 refines scales after GPTQ via coordinate descent with a closed-form update, incorporating quantization-error propagation from preceding layers. The method yields closed-form updates for first-layer scales and a refined rule for subsequent layers that accounts for input deviation through $\mathbf{R} = \mathbb{E}[\Delta \mathbf{X} \mathbf{X}^T]$, improving layer-wise reconstruction accuracy with negligible overhead. Experiments on Llama-based models demonstrate consistent gains over GPTQ in low-bit regimes across multiple group configurations, enabling more accurate, efficient deployment of large language models.

Abstract

Group-wise quantization is an effective strategy for mitigating accuracy degradation in low-bit quantization of large language models (LLMs). Among existing methods, GPTQ has been widely adopted due to its efficiency; however, it neglects input statistics and inter-group correlations when determining group scales, leading to a mismatch with its goal of minimizing layer-wise reconstruction loss. In this work, we propose a two-stage optimization framework for group scales that explicitly minimizes the layer-wise reconstruction loss. In the first stage, performed prior to GPTQ, we initialize each group scale to minimize the group-wise reconstruction loss, thereby incorporating input statistics. In the second stage, we freeze the integer weights obtained via GPTQ and refine the group scales to minimize the layer-wise reconstruction loss. To this end, we employ the coordinate descent algorithm and derive a closed-form update rule, which enables efficient refinement without costly numerical optimization. Notably, our derivation incorporates the quantization errors from preceding layers to prevent error accumulation. Experimental results demonstrate that our method consistently enhances group-wise quantization, achieving higher accuracy with negligible overhead.

Figures (1)

• Figure 1: Illustration of one output channel in a weight matrix and Hessian for group-wise quantization