Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs
Davide Paglieri, Saurabh Dash, Tim Rocktäschel, Jack Parker-Holder
TL;DR
This work investigates how calibration data and activation outliers influence one-shot post-training quantization (PTQ) across modern open-source LLMs. It benchmarks six models (including OPT 6.7B and Llama-2/3, Mistral 7B, Command-R 35B) with GPTQ, AWQ, and SmoothQuant using diverse calibration sets (RedPajama, random punctuation, ARC-Challenge, PiQa, FLORES+-languages) and evaluates perplexity and downstream tasks. The main finding is that contemporary models exhibit strong robustness to calibration-set quality, content, and language, while OPT 6.7B remains sensitive to outliers. This suggests a paradigm shift in PTQ research, moving toward inference-speed optimization and end-to-end weight-and-activation quantization for state-of-the-art LLMs, rather than preserving outlier activations, driven by advances in training and model architectures.
Abstract
Post-Training Quantization (PTQ) enhances the efficiency of Large Language Models (LLMs) by enabling faster operation and compatibility with more accessible hardware through reduced memory usage, at the cost of small performance drops. We explore the role of calibration sets in PTQ, specifically their effect on hidden activations in various notable open-source LLMs. Calibration sets are crucial for evaluating activation magnitudes and identifying outliers, which can distort the quantization range and negatively impact performance. Our analysis reveals a marked contrast in quantization effectiveness across models. The older OPT model, upon which much of the quantization literature is based, shows significant performance deterioration and high susceptibility to outliers with varying calibration sets. In contrast, newer models like Llama-2 7B, Llama-3 8B, Command-R 35B, and Mistral 7B demonstrate strong robustness, with Mistral 7B showing near-immunity to outliers and stable activations. These findings suggest a shift in PTQ strategies might be needed. As advancements in pre-training methods reduce the relevance of outliers, there is an emerging need to reassess the fundamentals of current quantization literature. The emphasis should pivot towards optimizing inference speed, rather than primarily focusing on outlier preservation, to align with the evolving characteristics of state-of-the-art LLMs.