Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats

Manyi Zhang, Ji-Fu Li, Zhongao Sun, Haoli Bai, Hui-Ling Zhen, Zhenhua Dong, Xianzhi Yu

TL;DR

This work tackles the problem of post-training quantization for Microscaling Floating-Point (MXFP) formats in large language models. It conducts a comprehensive empirical study across MXFP8 and MXFP4, evaluating more than seven PTQ algorithms on 15 benchmarks and three LLM families, using four quantization paradigms. Key findings show MXFP8 enables near-lossless deployment, while MXFP4 remains challenging and highly sensitive to scaling factors; error compensation and affine methods generally perform best, with RTN serving as a strong baseline. The results offer practical guidance for MXFP-aware quantization design and highlight that MXFP is a distinct numerical regime that benefits from format-specific optimization, particularly in multimodal models where the LLM dominates sensitivity and visual tokens exhibit robust behavior under MXFP.

Abstract

Microscaling Floating-Point (MXFP) has emerged as a promising low-precision format for large language models (LLMs). Despite various post-training quantization (PTQ) algorithms being proposed, they mostly focus on integer quantization, while their applicability and behavior under MXFP formats remain largely unexplored. To address this gap, this work conducts a systematic investigation of PTQ under MXFP formats, encompassing over 7 PTQ algorithms, 15 evaluation benchmarks, and 3 LLM families. The key findings include: 1) MXFP8 consistently achieves near-lossless performance, while MXFP4 introduces substantial accuracy degradation and remains challenging; 2) PTQ effectiveness under MXFP depends strongly on format compatibility, with some algorithmic paradigms being consistently more effective than others; 3) PTQ performance exhibits highly consistent trends across model families and modalities, in particular, quantization sensitivity is dominated by the language model rather than the vision encoder in multimodal LLMs; 4) The scaling factor of quantization is a critical error source in MXFP4, and a simple pre-scale optimization strategy can significantly mitigate its impact. Together, these results provide practical guidance on adapting existing PTQ methods to MXFP quantization.

Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats

TL;DR

This work tackles the problem of post-training quantization for Microscaling Floating-Point (MXFP) formats in large language models. It conducts a comprehensive empirical study across MXFP8 and MXFP4, evaluating more than seven PTQ algorithms on 15 benchmarks and three LLM families, using four quantization paradigms. Key findings show MXFP8 enables near-lossless deployment, while MXFP4 remains challenging and highly sensitive to scaling factors; error compensation and affine methods generally perform best, with RTN serving as a strong baseline. The results offer practical guidance for MXFP-aware quantization design and highlight that MXFP is a distinct numerical regime that benefits from format-specific optimization, particularly in multimodal models where the LLM dominates sensitivity and visual tokens exhibit robust behavior under MXFP.

Abstract

Microscaling Floating-Point (MXFP) has emerged as a promising low-precision format for large language models (LLMs). Despite various post-training quantization (PTQ) algorithms being proposed, they mostly focus on integer quantization, while their applicability and behavior under MXFP formats remain largely unexplored. To address this gap, this work conducts a systematic investigation of PTQ under MXFP formats, encompassing over 7 PTQ algorithms, 15 evaluation benchmarks, and 3 LLM families. The key findings include: 1) MXFP8 consistently achieves near-lossless performance, while MXFP4 introduces substantial accuracy degradation and remains challenging; 2) PTQ effectiveness under MXFP depends strongly on format compatibility, with some algorithmic paradigms being consistently more effective than others; 3) PTQ performance exhibits highly consistent trends across model families and modalities, in particular, quantization sensitivity is dominated by the language model rather than the vision encoder in multimodal LLMs; 4) The scaling factor of quantization is a critical error source in MXFP4, and a simple pre-scale optimization strategy can significantly mitigate its impact. Together, these results provide practical guidance on adapting existing PTQ methods to MXFP quantization.
Paper Structure (44 sections, 2 equations, 5 figures, 10 tables)

This paper contains 44 sections, 2 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Low-Bit Integer (INT) and Floating-Point (FP) Quantization
  4. Microscaling Floating Point (MXFP) Quantization
  5. Post-training Quantization Methods
  6. Channel-Wise Transformation.
  7. Error Compensation.
  8. Rotational Transformation.
  9. Affine Transformation.
  10. Evaluations
  11. Setup
  12. Quantization Settings.
  13. Evaluation Benchmarks.
  14. Evaluation Models.
  15. Performance on Different MXFP Quantization Settings (RQ1)
  16. ...and 29 more sections

Figures (5)

  • Figure 1: The overview of our empirical evaluations.
  • Figure 2: Average accuracy of Reasoning tasks on openPangu-Embedded-7B-V1.1.
  • Figure 3: Recovery rate performance of various PTQ methods across MXFP quantization settings. Each curve represents results from all quantization settings of a model. We also show the pairwise correlation of two different models. The correlation is calculated using the Pearson correlation coefficient.
  • Figure 4: Impact of scaling factor error. Restoring low-precision values and low-precision scales (LPX, LPS) to low-precision values with high-precision scales (LPX, HPS) leads to a clear reduction in perplexity (PPL).
  • Figure 5: Activation distributions of the q_proj module in layer 8 of openPangu-Embedded-7B-V1.1 with different quantization methods. Each subplot shows the activations observed during inference, highlighting how quantization methods alter the dynamic range and distribution.