Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models

TL;DR

This work tackles the difficulty of post-training quantization for large language models by identifying that high-impact parameters drive quantization errors, especially at very low bit-widths. It introduces a layer-wise quadratic optimization framework to determine the optimal ratio of high-impact parameters per layer, incorporating inter-layer dependencies and a block-wise Hessian approximation. A hybrid quantization strategy follows: high-impact parameters are quantized with AdaRound (via learnable rounding) while the remaining parameters use efficient methods like weight clipping, enabling better accuracy under tight bit budgets. Empirical results on LLaMA-2 models demonstrate substantial improvements in perplexity on generation datasets and higher zero-shot task accuracy compared with state-of-the-art PTQ methods, particularly in 2-bit and 3-bit regimes. The approach offers a scalable, hardware-friendly path to deploying large models with minimal degradation in performance.

Abstract

Large language models (LLMs) have significantly advanced natural language processing, but their massive parameter counts create substantial computational and memory challenges during deployment. Post-training quantization (PTQ) has emerged as a promising approach to mitigate these challenges with minimal overhead. While existing PTQ methods can effectively quantize LLMs, they experience substantial accuracy loss at extremely low bit-widths, primarily due to high-impact parameters that significantly influence quantization performance. Several approaches address these issues by identifying and retaining the high-impact parameters in FP16 format. However, they apply fixed ratios of high-impact parameters across all layers, overlooking layer-wise sensitivity variations. In this paper, we propose a quadratic optimization framework that determines layer-specific ratios of high-impact parameters while considering inter-layer dependencies. We quantize high-impact parameters to moderate bit-widths, which often result in negligible performance degradation in quantized LLMs, while the remaining parameters can be quantized to extremely low bit-widths. Under the same resource-constrained budget, this allows for preserving more high-impact parameters than methods that keep selecting a few in FP16 format. Additionally, the proposed framework allows us to leverage an advanced quantization method that often requires extensive learnable parameters solely for high-impact parameters, while applying a computationally efficient method to the rest. Our approach achieves an effective balance between computational efficiency and model accuracy while maintaining high performance compared to state-of-the-art methods.

Figures (2)

• Figure 1: Distribution of average Fisher information across channel dimensions at several layers in the LLaMA-2-7B model. There is a significant difference in importance scores for each layer, indicating that impactful parameters are concentrated in a subset of channels. Additionally, the ratio of high-impact parameters differs across layers.
• Figure 2: High-impact parameter ratio for selected layers across blocks in the LLaMA-2-7B model with W3A16 quantization.