Is Finer Better? The Limits of Microscaling Formats in Large Language Models

TL;DR

Is Finer Better? The Limits of Microscaling Quant formats investigates whether decreasing per-block sizes in microscale quantization always improves accuracy for large language models. The authors observe a surprising anomaly: with FP8 UE4M3 scales, smaller blocks can increase quantization error for narrow weight distributions, contradicting intuition, and they develop a theoretical framework that decouples error sources tied to $x_{ ext{max}}$, scale nonzero, and scale zero. This framework aligns with both pretrained-model distributions and ideal Normal distributions, validating the core mechanism behind the inversion and enabling cross-format generalization. To mitigate the effect, they propose FP8 unsigned E5M3 (UE5M3) scales, which extend dynamic range with minimal hardware impact and match or exceed UE4M3 performance without per-tensor scaling across architectures. The work provides practical guidance for robust microscale quantization in future LLM accelerators, balancing memory, compute, and accuracy.

Abstract

Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.

Figures (17)

• Figure 1: (a) FP4 microscaling quantization using BF16 scales. (b) Impact of FP8 UE4M3 scales. Anomalous data points showcasing perplexity inversion have been circled.
• Figure 2: (a) Per-block MSE of the first Query weight tensor, block size (bs) 8 vs 16. (b) Per-tensor MSE vs standard deviation $\sigma$ of each weight tensor of granite-3.3-8b and llama-2-7b. Tensors quantized as FP4 using FP8 UE4M3 scales, with bs 8 or 16. (c) MSE vs $\sigma$ using BF16 scales instead.
• Figure 3: (a) MSE - $\sigma$ dependency for weights of 3 pre-trained models (dots) against a Normal distribution (black line). (b) Behavior of several ideal distributions. (c) Normal distribution compared to theoretical results, and decomposition of 3 contributions to the theoretical error.
• Figure 4: (a) Details of UE5M3 hardware implementation. (b,c) Perplexity vs block size using microscaling FP4 with FP8 UE5M3 scales.
• Figure 5: (a) Perplexity gap vs. block size for FP4 with FP8-UE4M3 scales quantization. Plotted with a logarithmic y-axis to account for the significant perplexity inversion observed in some models. (b) Even when perplexity inversion is not present down to block size 8 (as llama-2-7b in Fig. \ref{['fig:ppl_vs_bs']}(b)), it can still emerge at even smaller block sizes (2, 4).