CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
Ziteng Sun, Adrian Benton, Samuel Kushnir, Asher Trockman, Vikas Singh, Suhas Diggavi, Ananda Theertha Suresh
TL;DR
This work tackles calibration-free post-training quantization for large language models by learning weight-transformations and adaptive rounding to minimize quantization loss without calibration data. It introduces a proxy Frobenius-norm objective and two classes of learnable transforms: block-diagonal matrices for single weight matrices and arbitrary transformations for coupled matrix pairs, plus a joint/iterative quantization scheme to reduce matrix-product errors. Empirical results on Gemma-2 models show consistent improvements over calibration-free baselines, achieving notable gains for 4-bit and 3-bit quantization with less than 3% extra computation and competitive performance relative to GPTQ. The approach offers a privacy-preserving, data-independent PTQ pathway with practical impact for deploying quantized LLMs in real-world, regulation-sensitive settings, and points to fruitful future work in combining with calibration-based techniques.
Abstract
Post-training quantization is an effective method for reducing the serving cost of large language models, where the standard approach is to use a round-to-nearest quantization level scheme. However, this often introduces large errors due to outliers in the weights. Proposed mitigation mechanisms include applying adaptive rounding, random rotation transformations or committing to a post-training target using calibration data. Unfortunately, this reliance on calibration data can be severely limiting in some real-world scenarios as such data may be unavailable or subject to privacy regulations. In this paper, we propose algorithms to optimize transformations and adaptive rounding without access to any calibration data. The optimization is achieved by designing a suitable proxy function for the quantization loss without calibration data. To maintain inference efficiency, we perform structured matrix transformations for single matrices. For paired weights that interact directly in the computation graph, we use dual matrix transformations and adaptive rounding methods. We conduct experiments on Gemma 2 models, and observe consistent improvement over the baselines. For Gemma 2 9B quantization, our method improves the average benchmark score from 61.9 to 62.4 for 4-bit quantization and from 52.0 to 60.6 for 3-bit quantization, while adding less than 3% of computation overhead. Furthermore, our method achieves performance comparable to the commonly used GPTQ method, which requires calibration data.