FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design

TL;DR

FlexQ tackles the practical deployment challenge of INT6 quantization for large language models by uniting a fine-grained, calibration-free quantization scheme with a BTC-based GPU kernel. The approach maintains near-FP16 accuracy while delivering significant end-to-end speedups and memory savings over state-of-the-art baselines. Key innovations include adaptive 6-bit weight quantization with selectively higher-precision activations in sensitive layers and a bit-level W6Ax kernel that eliminates padding inefficiencies via BTC-based computations and memory-aware data layouts. Empirical results on LLaMA and OPT models demonstrate substantial improvements in both accuracy and performance, making INT6 quantization a viable option for practical LLM serving; code is released for public use.

Abstract

Large Language Models (LLMs) demonstrate exceptional performance but entail significant memory and computational costs, restricting their practical deployment. While existing INT4/INT8 quantization reduces these costs, they often degrade accuracy or lack optimal efficiency. INT6 quantization offers a superior trade-off between model accuracy and inference efficiency, but lacks hardware support in modern GPUs, forcing emulation via higher-precision arithmetic units that limit acceleration. In this paper, we propose FlexQ, a novel post-training INT6 quantization framework combining algorithmic innovation with system-level optimizations. FlexQ employs uniform 6-bit weight quantization across all layers, with adaptive retention of 8-bit activations in layers identified through layer-wise sensitivity analysis. To maximize hardware efficiency, we develop a specialized high-performance GPU kernel supporting matrix multiplication for W6A6 and W6A8 representations via Binary Tensor Core (BTC) equivalents, effectively bypassing the lack of native INT6 tensor cores. Evaluations on LLaMA family models show FlexQ maintains near-FP16 accuracy, with perplexity increases of no more than 0.1 on WikiText2. The proposed kernel achieves an average 1.39$\times$ speedup over ABQ-LLM on LLaMA-2-70B linear layers. End-to-end, FlexQ delivers 1.33$\times$ inference acceleration and 1.21$\times$ memory savings over SmoothQuant. Code is released at https://github.com/FlyFoxPlayer/FlexQ.

Figures (12)

• Figure 1: Illustration of LLM inference.
• Figure 2: The input activation distribution in the LLaMA-7B model displays pronounced outliers across all token sequences, particularly within the k_proj of layer 3 and the up_proj of layer 14.
• Figure 3: Memory access analysis of data loading from shared memory to registers on a per-thread basis. Here, E represents a weight element, W denotes a 32-bit word, and T indicates a thread.
• Figure 4: Perplexity results (lower is better) for the LLaMA-7B and LLaMA-2-7B models, evaluating various linear layers on the WikiText2 dataset.
• Figure 5: Overview of the FlexQ workflow for LLaMA models, where ReQuant and DeQuant denote online quantization and dequantization operations, respectively, and BitPacking represents the online Bit-level Data Packing operation for activations.