SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs

TL;DR

The paper introduces SQ-format, a unified sparse-quantized data format designed to enable hardware-friendly hybrid-precision computation for LLMs. By bank-partitioning matrices and coupling high- and low-precision components, SQ-format achieves Pareto improvements in accuracy and throughput, outperforming standard W4A8 baselines and approaching W4A4 performance in many cases. It provides two PTQ algorithms (weights and activations), a static activation quantization strategy to avoid runtime overhead, and a hardware-algorithm co-design framework with guidance on bank size, sparsity, and precision choices. Experimental results across multiple models demonstrate significant throughput gains with minimal accuracy loss, supported by hardware-synthesis evidence and end-to-end profiling. The work offers a practical path toward deploying dense and mixture-of-experts LLMs on current GPUs and next-generation AI accelerators through a co-designed data representation and processing pipeline.

Abstract

Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.

Figures (7)

• Figure 1: An example of a weight matrix using SQ-format ($h_\text{high}=\text{INT8}$, $h_\text{low}=\text{INT4}$, $s=0.5$). In this case, each column of each bank is split into a high-precision part and a low-precision part for quantization as a group. The high-precision part is stored compactly, while the low-precision part is still stored in its original shape, but the corresponding high-precision places is valued $v_\text{mask}=15$.
• Figure 2: Hardware implementation for applying SQ-format on weights and activations.
• Figure 3: Throughput of static SQ-format on activations from $s=0.5$ (2x sparse) to $0.9375$ (16x sparse).
• Figure 5: Accuracy-Speed Pareto frontier on Llama-3 models.
• Figure 6: Activation partition results from Layer 30 of Llama-3-8B. $\text{bank\_size}=32$ provides more flexible precision partition than $\text{bank\_size}=4$.