FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs

TL;DR

This work tackles the memory and accuracy trade-offs of quantizing large language models to ultra-low bit-widths. It introduces FineQ, a software-hardware co-design that uses fine-grained intra-cluster quantization with 3-bit outlier protection and a memory-aligned encoding scheme, paired with a temporal-coding accelerator to simplify multipliers. Empirical results show improved perplexity over state-of-the-art mixed-precision quantization at similar average bit-widths, and hardware evaluation reports up to 1.79× energy efficiency and a 61.2% area reduction. The approach enables more practical deployment of large transformers on edge devices by balancing model fidelity and hardware efficiency.

Abstract

Large language models (LLMs) have significantly advanced the natural language processing paradigm but impose substantial demands on memory and computational resources. Quantization is one of the most effective ways to reduce memory consumption of LLMs. However, advanced single-precision quantization methods experience significant accuracy degradation when quantizing to ultra-low bits. Existing mixed-precision quantization methods are quantized by groups with coarse granularity. Employing high precision for group data leads to substantial memory overhead, whereas low precision severely impacts model accuracy. To address this issue, we propose FineQ, software-hardware co-design for low-bit fine-grained mixed-precision quantization of LLMs. First, FineQ partitions the weights into finer-grained clusters and considers the distribution of outliers within these clusters, thus achieving a balance between model accuracy and memory overhead. Then, we propose an outlier protection mechanism within clusters that uses 3 bits to represent outliers and introduce an encoding scheme for index and data concatenation to enable aligned memory access. Finally, we introduce an accelerator utilizing temporal coding that effectively supports the quantization algorithm while simplifying the multipliers in the systolic array. FineQ achieves higher model accuracy compared to the SOTA mixed-precision quantization algorithm at a close average bit-width. Meanwhile, the accelerator achieves up to 1.79x energy efficiency and reduces the area of the systolic array by 61.2%.

Figures (9)

• Figure 1: Perplexity of LLaMA-2-7B on C4 under differnet bit-widths. Lower perplexity means better model accuracy. Round-to-nearest (RTN), GPTQ, PB-LLM (10% weight of FP16), and OWQ(g=128) suffer from accuracy loss at ultra-low bits. FineQ demonstrates better performance than these methods.
• Figure 2: (a) The transformer block architecture. (b) Memory layout for serving a 13B-parameter LLM on the NVIDIA A100 (40GB)kwon2023efficient.
• Figure 3: Three observations about hardware-software co-design of mixed-precision quantization.
• Figure 4: An example of fine-grained intra-cluster quantization algorithm.
• Figure 5: (a) Overview of FineQ architecture. (b) Temporal coding PE array design. (c) Temporal Encoder, PE, and ACC design.