A Comprehensive Evaluation of Quantization Strategies for Large Language Models
Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, Deyi Xiong
TL;DR
This work introduces a structured three‑dimensional framework (knowledge & capacity, alignment, efficiency) for evaluating quantized instruction‑tuned LLMs across ten diverse benchmarks. It systematically compares three PTQ quantization strategies (LLM.int8(), GPTQ, SpQR) on Qwen‑Chat models (7B, 14B, 72B), revealing that 4‑bit quantization largely preserves non‑quantized performance while 3‑bit and below cause significant degradation. The study identifies perplexity as a strong proxy for quantized LLM performance on most benchmarks and demonstrates that outlier isolation in SpQR enables effective 2‑bit quantization, albeit with substantial engineering and hardware requirements. These findings inform deployment decisions in resource‑constrained settings, emphasizing that larger models with moderate quantization can offer favorable trade‑offs, while recognizing the need for hardware acceleration and careful benchmarking to achieve practical speed and memory efficiency. Limitations include benchmark contamination risks and focus on a single model family, suggesting broad future work on diverse architectures and deployment environments.
Abstract
Increasing the number of parameters in large language models (LLMs) usually improves performance in downstream tasks but raises compute and memory costs, making deployment difficult in resource-limited settings. Quantization techniques, which reduce the bits needed for model weights or activations with minimal performance loss, have become popular due to the rise of LLMs. However, most quantization studies use pre-trained LLMs, and the impact of quantization on instruction-tuned LLMs and the relationship between perplexity and benchmark performance of quantized LLMs are not well understood. Evaluation of quantized LLMs is often limited to language modeling and a few classification tasks, leaving their performance on other benchmarks unclear. To address these gaps, we propose a structured evaluation framework consisting of three critical dimensions: (1) knowledge \& capacity, (2) alignment, and (3) efficiency, and conduct extensive experiments across ten diverse benchmarks. Our experimental results indicate that LLMs with 4-bit quantization can retain performance comparable to their non-quantized counterparts, and perplexity can serve as a proxy metric for quantized LLMs on most benchmarks. Furthermore, quantized LLMs with larger parameter scales can outperform smaller LLMs. Despite the memory savings achieved through quantization, it can also slow down the inference speed of LLMs. Consequently, substantial engineering efforts and hardware support are imperative to achieve a balanced optimization of decoding speed and memory consumption in the context of quantized LLMs.