LLM-FP4: 4-Bit Floating-Point Quantized Transformers

TL;DR

The paper tackles the challenge of post-training quantization for transformers at ultra-low bit-width by advocating 4-bit floating-point quantization for weights and activations. It introduces FPQ, a search-based baseline to select FP formats and clipping ranges, and complements it with a per-channel activation strategy via pre-shifted exponent bias to mitigate inter-channel variance. The approach delivers state-of-the-art performance at 6/8 bits and enables practical 4-bit quantization for LLaMA-13B, BERT, and ViT, with strong zero-shot reasoning and GLUE/ImageNet results. The work also demonstrates favorable hardware cost implications, suggesting FPQ is scalable to real-world deployments on FP-enabled hardware.

Abstract

We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values, in a post-training manner. Existing post-training quantization (PTQ) solutions are primarily integer-based and struggle with bit widths below 8 bits. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail or bell-shaped distributions, and it has emerged as a default choice in many hardware platforms. One characteristic of FP quantization is that its performance largely depends on the choice of exponent bits and clipping range. In this regard, we construct a strong FP-PTQ baseline by searching for the optimal quantization parameters. Furthermore, we observe a high inter-channel variance and low intra-channel variance pattern in activation distributions, which adds activation quantization difficulty. We recognize this pattern to be consistent across a spectrum of transformer models designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models. To tackle this, we propose per-channel activation quantization and show that these additional scaling factors can be reparameterized as exponential biases of weights, incurring a negligible cost. Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63.1 on the common sense zero-shot reasoning tasks, which is only 5.8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12.7 points. Code is available at: https://github.com/nbasyl/LLM-FP4.

Figures (7)

• Figure 1: An illustration of floating-point (FP) quantization process using FP5 (E2M2) positive axis. The real-valued clipped $X_{\rm R}"$ in Eq. \ref{['eq:fp_clip']} is rescaled by the real-valued scaling factor $\tilde{\alpha}$. Then, the quantization step-size $v$ is determined by the range $[2^p, 2^p+1)$ in which $\frac{X_{\rm R}"}{\tilde{\alpha}}$ falls (Eq. \ref{['eq:fp_quant_scale']}). Here, $p \in \{0,1,...,2^{e-1}\}$ is the exponent bit value. Lastly, $X$ can be quantized to low-bit floating point values simply by $X_{\rm{FP}} = \tilde{\alpha} \cdot v \cdot \left\lfloor \frac{X_{\rm R}"}{\tilde{\alpha} \cdot v} \right\rceil$ (Eq. \ref{['eq:fp_quant']}).
• Figure 2: Magnitude of the output activations of the feed-forward network blocks in LLaMA-7B, BERT, and DeiT.
• Figure 3: Overview of pre-shifted exponent bias method: (a) Search phase: The real-valued channel-wise scaling exponent bias for activations ($\tilde{\mathbf{b}}_j$) is partitioned into a real-valued tensor-wise exponent bias ($\rho$), and the integer-based channel-wise exponent bias ($\tilde{\mathbf{b}}^{ori}_j$). (b) Reparameterization and weight pre-computation: Once the optimal values are determined on the calibration set, $\tilde{\mathbf{b}}^{ori}_j$ are re-parameterized into the weight tensor. The weights are pre-computed to apply the bias, therefore this is a one-time cost. (c) Inference phase: The method leverages efficient matrix multiplication between low-bit floating-point matrices.
• Figure 4: Quantization error of different formats for BERT layers.
• Figure 5: Magnitude of the output activations of different modules in BERT (left column), and DeiT-S (right column).