The Power of Negative Zero: Datatype Customization for Quantized Large Language Models

TL;DR

This work tackles the memory and compute bottlenecks of large language models by enhancing post-training quantization through a floating-point datatype customization. The authors propose RaZeR, which remaps the redundant FP negative-zero encoding in FP4/FP3 to a small set of special values, enabling an extra quantization level that better fits LLM value distributions at 3–4 bits while preserving compute efficiency. Across weight-only and KV-cache quantization, RaZeR yields consistent accuracy improvements and substantial speedups on GPUs, including up to $2.72\times$ end-to-end decoding throughput and $7.56\times$ GEMV speedups, largely due to compact encoding and a fast CUDA kernel. The results suggest RaZeR is a practical, hardware-friendly approach for deploying quantized LLMs on modern GPUs, with potential gains from future hardware support for special-value decoding and mixed-precision tensor cores.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks, quickly becoming one of the most prevalent AI workloads. Yet the substantial memory requirement of LLMs significantly hinders their deployment for end users. Post-training quantization (PTQ) serves as one of the most hardware-efficient methods to mitigate the memory and computational demands of LLMs. Although the traditional integer (INT) datatype has received widespread adoption in PTQ methods, floating-point (FP) quantization has emerged as a viable alternative thanks to its effectiveness in fitting LLM numerical distributions. However, the FP datatype in sign-magnitude binary representation contains both positive and negative zero, which constrains its representation capability, particularly under low precision (3 and 4 bits). In this paper, we extend the basic FP datatype to perform Redundant Zero Remapping (RaZeR), which remaps the negative zero FP encoding to a set of pre-defined special values to maximally utilize FP quantization encodings and to better fit LLM numerical distributions. Through careful selection of special values, RaZeR outperforms conventional asymmetric INT quantization while achieving high computational efficiency. We demonstrate that RaZeR can be seamlessly integrated with quantization algorithms for both weights and KV-cache, including advanced methods with clipping and transformations, and consistently achieve better model accuracy. Additionally, we implement a fast GEMV kernel with fused dequantization that efficiently converts the 4-bit RaZeR value to FP16 through novel bit-level manipulation. On modern GPUs, our evaluation shows that RaZeR improves the GEMV speed by up to 7.56$\times$ compared to the FP16 implementation, while achieving up to 2.72$\times$ speedup in the LLM decoding throughput.

Figures (5)

• Figure 1: Floating point (FP) datatypes have redunant encodings for the zero value as it can be represented either as $+0$ or $-0$. We remap the redundant zero to a learnable "special value" per quantization group to significantly decrease the LLM quantization error at 3 and 4 bits, for both weight and KV-cache quantization.
• Figure 2: Numerical distribution and corresponding quantization error for quantizing a vector from the KV-Cache of Llama-3.2-3B using three datatypes. Compared to INT, FP decreases the error at the center of the distribution where there is the highest density of numbers, at the cost of higher error at the tail of the distribution. RaZeR further improves the FP datatype by remapping $-0$ to an additional "special value" thus decreasing the approximation error in its vicinity. In 3-bit RaZeR, one configuration (left) benefits more from extra range, while the other (right) benefits from additional resolution within the same range, both resulting in reduced overall quantization error.
• Figure 3: Normalized weight quantization error ($\downarrow$) with respect to different special values ($SV{}$) for FP3. We use group-wise quantization with a group size of 128.
• Figure 4: End-to-end throughput of RaZeR compared to FP16 and Marlin.
• Figure 5: GEMV latency on RTX 4090. We compare the latency for linear layers of Llama-2 models. The horizontal-axis labels represent the weight matrix size in different linear layers.