Mixed-Precision Quantization for Language Models: Techniques and Prospects

TL;DR

This paper surveys mixed-precision quantization for large language models, formalizing inter- and intra-layer bitwidth allocation and focusing on weights, activations, and KV caches. It categorizes frameworks into mixed-precision weight, weight-activation, and fully mixed schemes, detailing methods such as GPTQ-based OBS, AWQ/SmoothQuant ET, and rotation-based QuaRot/quIP, plus hardware-driven approaches like MPMLC and HOBBIT. A comparative analysis highlights near-baseline perplexities for many 4-bit configurations, with more aggressive 2–3 bit schemes achieving variable success depending on model and task; it also discusses hardware implications and deployment trade-offs. Finally, the authors outline open issues and future directions, emphasizing hardware/software co-design, activation/KV quantization challenges, microscaling formats, and scalable optimization strategies for billion-parameter LMs, underscoring the practical impact for efficient, scalable LM deployment across cloud and edge environments.

Abstract

The rapid scaling of language models (LMs) has resulted in unprecedented computational, memory, and energy requirements, making their training and deployment increasingly unsustainable. Quantization has emerged as an essential compression technique to reduce model size, alleviate memory bottlenecks, and accelerate inference. However, while uniform low-bit quantization (e.g., INT8, INT4) provides significant efficiency gains, it can degrade accuracy in sensitive components of transformer-based LMs. Mixed-precision quantization offers a promising alternative by selectively allocating precision across layers or within tensors to balance efficiency and accuracy. This survey provides a comprehensive overview of Mixed-Precision quantization frameworks for LMs (MXPLMs). We first review quantization fundamentals, including uniform and non-uniform quantizers, quantization granularity, and methods widely used in post-training quantization. We then categorize and compare recent MXPLM frameworks according to their bit allocation strategies and precision configurations across weights, activations, and key-value caches. A comparative analysis highlights differences in perplexity, zero-shot task performance, and deployment trade-offs. Furthermore, we contrast MXPLMs with earlier mixed-precision quantization methods for deep neural networks, identifying strategies that transfer and those that face challenges in the LM setting. Finally, we summarize open issues and future directions, including hardware-aware design, activation quantization, and scalable optimization methods for billion-parameter models. By consolidating recent advances, this work serves as a reference for understanding the current landscape and research prospects of mixed-precision quantization for large-scale language models.

Figures (6)

• Figure 1: ELO scores of state-of-the-art LMs released over the last three years. Scores are obtained from chiang2024chatbot, where the circle size represents the total number of parameters in a model. For models whose sizes were not officially disclosed, the values shown reflect the average of multiple open-source estimates.
• Figure 2: Illustration of different quantization schemes using 8-bit representation.
• Figure 3: Taxonomy of mixed-precision methods based on the quantization methods they employ.
• Figure 4: Overview of mixed-precision LM (MXPLM) frameworks; UPW/UPA: uniform-precision weights/activations, MPW/MPA: mixed-precision weights/activations.
• Figure 5: Perplexity of MXPLM frameworks reported on (a) LLaMA2-13B and (b) LLaMA3-8B using WikiText2 for different average precisions. Note: "(x%)" indicates sparsity of $x\%$.