Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization

Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng, Tianchi Hu, Guanglin Li, Guipeng Hu, Junsong Wang, Jingwen Leng

TL;DR

This work tackles the accuracy gap in 4-bit MX quantization for large language models by introducing M$^{\text{2}}$XFP, a metadata-augmented microscaling format that uses a hybrid allocation of element-level extra mantissa for activations and subgroup-level mantissa refinement with adaptive scales for weights. It provides an EBW-aware design-space exploration, identifies an asymmetry between weights and activations, and implements lightweight hardware support integrated into a systolic-array accelerator. Empirical results show near-FP16 fidelity at an effective 4.5-bit precision, achieving up to 70.6% reduction in accuracy loss over MXFP4 and 37.3% over NVFP4, along with up to 1.91x speedup and 1.75x energy savings. The approach demonstrates that carefully designed metadata can bridge MX efficiency with the fidelity requirements of modern LLM quantization, offering a practical path toward scalable, low-bit inference.

Abstract

Existing low-bit Microscaling (MX) formats, such as MXFP4, often suffer from substantial accuracy degradation due to the use of a shared scaling factor with the Power-of-Two format. In this work, we explore strategies that introduce minimal metadata to recover accuracy lost during quantization while maintaining high bit efficiency across a wide range of large language models. We propose a complete algorithm-hardware co-design based on flexible metadata, featuring an online quantization with simple encoding. To support the proposed method efficiently, we implement a lightweight hardware unit and integrate it into the accelerator. Evaluation results demonstrate that our method substantially narrows the accuracy gap, achieving on average a 70.63% reduction in accuracy loss compared to MXFP4 and a 37.30% reduction relative to the latest NVFP4 on LLM benchmarks. Furthermore, our design delivers up to 1.91$\times$ speedup and 1.75$\times$ energy savings over state-of-the-art accelerators. Our code is available at https://github.com/SJTU-ReArch-Group/M2XFP_ASPLOS26.

M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization

TL;DR

This work tackles the accuracy gap in 4-bit MX quantization for large language models by introducing MXFP, a metadata-augmented microscaling format that uses a hybrid allocation of element-level extra mantissa for activations and subgroup-level mantissa refinement with adaptive scales for weights. It provides an EBW-aware design-space exploration, identifies an asymmetry between weights and activations, and implements lightweight hardware support integrated into a systolic-array accelerator. Empirical results show near-FP16 fidelity at an effective 4.5-bit precision, achieving up to 70.6% reduction in accuracy loss over MXFP4 and 37.3% over NVFP4, along with up to 1.91x speedup and 1.75x energy savings. The approach demonstrates that carefully designed metadata can bridge MX efficiency with the fidelity requirements of modern LLM quantization, offering a practical path toward scalable, low-bit inference.

Abstract

Existing low-bit Microscaling (MX) formats, such as MXFP4, often suffer from substantial accuracy degradation due to the use of a shared scaling factor with the Power-of-Two format. In this work, we explore strategies that introduce minimal metadata to recover accuracy lost during quantization while maintaining high bit efficiency across a wide range of large language models. We propose a complete algorithm-hardware co-design based on flexible metadata, featuring an online quantization with simple encoding. To support the proposed method efficiently, we implement a lightweight hardware unit and integrate it into the accelerator. Evaluation results demonstrate that our method substantially narrows the accuracy gap, achieving on average a 70.63% reduction in accuracy loss compared to MXFP4 and a 37.30% reduction relative to the latest NVFP4 on LLM benchmarks. Furthermore, our design delivers up to 1.91 speedup and 1.75 energy savings over state-of-the-art accelerators. Our code is available at https://github.com/SJTU-ReArch-Group/M2XFP_ASPLOS26.
Paper Structure (34 sections, 4 equations, 13 figures, 8 tables, 1 algorithm)

This paper contains 34 sections, 4 equations, 13 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Model Quantization
  4. Microscaling Data Format
  5. Motivation
  6. Analysis of MX Quantization Error
  7. A Taxonomy of Quantization Design Dimensions
  8. Scaling Factor: Converging Toward Group-Level E8M0/FP8
  9. Data Type: Expressive but Hardware-Prohibitive
  10. Metadata: A Flexible Yet Underutilized Design Axis
  11. Takeaway: Metadata as the Key Lever
  12. M$^{\text{2}}$XFP Analysis and Design
  13. Framework for Design Space Exploration
  14. Pareto-Optimal Analysis of Metadata Strategies
  15. Evaluation Method
  16. ...and 19 more sections

Figures (13)

  • Figure 1: Microscaling data format.
  • Figure 2: FP4 quantization: A comparison of FP16 and E8M0 scaling factors.
  • Figure 3: Perplexity of 4-bit quantization on LLaMA3, retaining the group-wise maximum in FP16 significantly enhances MXFP4.
  • Figure 4: Perplexity decreases with increasing equivalent bit width (EBW), but the improvement diminishes beyond g-32 despite larger bit wdiths.
  • Figure 5: MX format with subgroup-level metadata. The figure contrasts two strategies for allocating extra bits: (1) Elem-EM/EE extends individual elements within subgroups; (2) Sg-EM/EE augments the subgroup scale.
  • ...and 8 more figures