Channel-Wise Mixed-Precision Quantization for Large Language Models

TL;DR

This work tackles the memory burden of large language models on edge devices by introducing Channel-Wise Mixed-Precision Quantization (CMPQ), a post-training quantization framework that allocates per-channel bit-width based on activation distributions. CMPQ uses channel-wise non-uniform quantization with two outlier-protection mechanisms (activation-based and quantization-based) to preserve critical information, enabling effective fractional-bit quantization without retraining. Empirical results on OPT and LLaMA2 demonstrate strong performance, especially at 2–3 bits, and competitive latency with significantly reduced memory compared to gradient-based approaches like SqueezeLLM. The method is data-efficient and robust to calibration-set variations, making CMPQ a practical solution for deploying LLMs on devices with varying storage and compute constraints.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes. Weight-only quantization presents a promising solution to reduce the memory footprint of LLMs. However, existing approaches primarily focus on integer-bit quantization, limiting their adaptability to fractional-bit quantization tasks and preventing the full utilization of available storage space on devices. In this paper, we introduce Channel-Wise Mixed-Precision Quantization (CMPQ), a novel mixed-precision quantization method that allocates quantization precision in a channel-wise pattern based on activation distributions. By assigning different precision levels to different weight channels, CMPQ can adapt to any bit-width constraint. CMPQ employs a non-uniform quantization strategy and incorporates two outlier extraction techniques that collaboratively preserve the critical information, thereby minimizing the quantization loss. Experiments on different sizes of LLMs demonstrate that CMPQ not only enhances performance in integer-bit quantization tasks but also achieves significant performance gains with a modest increase in memory usage. CMPQ thus represents an adaptive and effective approach to LLM quantization, offering substantial benefits across diverse device capabilities.

Figures (5)

• Figure 1: Illustration of different quantization approaches under a fixed bit-width constraint, such as 3 bits. (a) Standard quantization methods focus on algorithmic optimization to improve model performance, quantizing all layers uniformly to 3 bits. (b) LLM-MQ li2023llm calculates layer-wise scores using first-order information and applies integer programming to assign lower bit-widths to less sensitive layers. (c) In contrast, our proposed CMPQ distributes the information loss evenly across layers by employing a channel-wise approach. This method assigns varying bit-widths within each layer based on activation distribution, ensuring that no single layer experiences significant information loss.
• Figure 2: Magnitude of absolute activation values of the self_attn.out_proj layer in the third layer of OPT-2.7B.
• Figure 3: Comparison of C4 perplexity between CMPQ and LLM-MQ for fractional bit-width quantization. The red star indicates the best performance achieved by integer-based baselines at {2, 3, and 4} bits.
• Figure 4: Outlier protection ratio and perplexity trade-off of 3-bit quantized OPT-6.7B model.
• Figure 5: Schematic of real quantization process of CMPQ. Given FP16 inputs and quantized weights, the features and weights are decomposed into sub-matrices based on the channel-wise precision allocation returned by Algorithm \ref{['alg:quantile']}. Then each pair of feature and weight sub-matrices are multiplied together. The outlier matrices are multiplied in FP16 with the original input. Finally, both outlier and regular outputs are accumulated in 16-bit floating point outputs.