QQQ: Quality Quattuor-Bit Quantization for Large Language Models

TL;DR

QQQ addresses the trade-off between accuracy and speed in W4A8 quantization for large language models by combining adaptive smoothing of activation channels with Hessian-based weight compensation. It also introduces specialized W4A8 GEMMs for per-channel and per-group weight quantization to maximize throughput, and demonstrates substantial speedups on vLLM-based inference. Experimental results show QQQ achieves competitive perplexities and zero-shot performance compared with state-of-the-art quantization methods while delivering speedups up to ~2x over FP16 and other baselines, especially at larger batch sizes and model scales. The work highlights how targeted activation smoothing, second-order weight adjustment, and hardware-aware GEMMs can enable efficient, high-accuracy 4-bit quantization for large language models.

Abstract

Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference. W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation. To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations. QQQ employs adaptive smoothing and Hessian-based compensation, significantly enhancing the performance of quantized models without extensive training. Furthermore, we meticulously engineer W4A8 GEMM kernels to increase inference speed. Our specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve impressive speed increases of 3.67$\times$ and 3.29 $\times$ over FP16 GEMM. Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 $\times$, 2.10$\times$, and 1.25$\times$ compared to FP16, W8A8, and W4A16, respectively.

Figures (8)

• Figure 1: Framework of QQQ.
• Figure 2: The W4A8 GEMM for per-channel weight quantization. $\mathbf{A}$ denotes the activation matrix. $\mathbf{W}$ denotes the weight matrix. $\mathbf{s_A}$ denotes the per-token quantization scale of activations. $\mathbf{s_W}$ denotes the per-channel quantization scale of weights. The superscript represents the data type. The green sections indicate data, and the orange sections indicate operations.
• Figure 3: The W4A8 GEMM for per-group weight quantization. $\mathbf{s_{W_c}}$ represents the per-channel quantization scale converting from dequantized FP16 weights back to INT8 weights. $\mathbf{s_{W_g}^{*}}$ represents the fused per-group scale converting from INT4 weights to INT8 weights.
• Figure 4: Design of FusedDequantQuant. $\mathbf{s_{W_g}}$ represents the per-group quantization scale converting from FP16 weights to INT4 weights. The orange sections indicate online operations, and the blue sections indicate offline operations.
• Figure 5: Design of FastFP16toINT8.