MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design

TL;DR

MixLLM introduces a global salience-guided mixed-precision quantization that assigns higher bit-width to salient output features across all layers, achieving superior accuracy with modest memory cost. By coupling a global precision search with a two-step dequantization pipeline and an optimized end-to-end GPU kernel, it attains state-of-the-art system efficiency while preserving or improving perplexity and downstream task performance. The approach synergizes algorithmic decisions (8-bit activations, 4-bit asymmetric weights in groups) with hardware-aware implementations to reduce dequantization overhead and maximize MatMul throughput. Empirical results across multiple models demonstrate notable speedups and accuracy gains over weight-only, weight-activation, and prior mixed-precision methods, validating MixLLM as a practical, scalable quantization solution for large-scale LLM deployment.

Abstract

Quantization has become one of the most effective methodologies to compress LLMs into smaller size. However, the existing quantization solutions still show limitations of either non-negligible accuracy drop or system inefficiency. In this paper, we make a comprehensive analysis of the general quantization principles on their effect to the triangle of accuracy, memory consumption and system efficiency. We propose MixLLM that explores the new optimization space of mixed-precision quantization between output features based on the insight that different output features matter differently in the model. MixLLM identifies the output features with high salience in the global view rather than within each single layer, effectively assigning the larger bit-width to output features that need it most to achieve good accuracy with low memory consumption. We present the sweet spot of quantization configuration of algorithm-system co-design that leads to high accuracy and system efficiency. To address the system challenge, we design the two-step dequantization to make use of the int8 Tensor Core easily and fast data type conversion to reduce dequantization overhead significantly, and present the software pipeline to overlap the memory access, dequantization and the MatMul to the best. Extensive experiments show that with only 10% more bits, the PPL increasement can be reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, while on average MMLU-Pro improves by 0.93 over the SOTA of three popular models. In addition to its superior accuracy, MixLLM also achieves state-of-the-art system efficiency.

Figures (6)

• Figure 1: Illustration of the quantization with mixed-precision between output features and kernel execution.
• Figure 2: The percentage of high-salient out features within each linear layer of Llama 3.1 8B model according to each feature's contribution to the final loss after quantizing to 4-bit, with 10% high-salient features globally. Each decoder layer contains q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj in order.
• Figure 3: The float and integer value of binary (010010110xx...x), each within a consecutive range.
• Figure 4: The GPU kernel software pipeline of group-wise W4A8/W8A8 quantized MatMul. It assumes perfect overlapping. G2S: load global to shared memory; S2R: load shared memory to register; MMA: matrix multiply-accumulation; I2F: integer to float conversion; deq: dequantize; acc: accumulate.
• Figure 5: The speedup of two types of single linear layers over torch float16 baseline on the A100 GPU.