MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving

TL;DR

The paper tackles the high cost of LLM serving by evaluating industry-driven block-based data formats and identifying outliers in ultra-low-bit blocks as a key bottleneck. It introduces MX+, a non-intrusive extension to the Microscaling (MX) formats that repurposes the BM element’s exponent as an extended mantissa to better represent outliers, enabling effective 4-bit MX for both weights and activations. Empirical results show MX+ significantly improves model accuracy (up to +42.15% over MXFP4 in some cases) with minimal overhead, and hardware-aware designs yield substantial end-to-end speedups while preserving accuracy. The work further demonstrates software and hardware integration strategies, including Tensor Core adaptations and channel-reordering techniques, broadening MX+’s applicability to non-FP formats and weight-only quantization scenarios. Overall, MX+ presents a practical, high-impact approach to efficient LLM inference in real-world systems.

Abstract

Reduced-precision data formats are crucial for cost-effective serving of large language models (LLMs). While numerous reduced-precision formats have been introduced thus far, they often require intrusive modifications to the software frameworks or are rather unconventional for widespread adoption across hardware vendors. In this paper, we instead focus on recent industry-driven variants of block floating-point (BFP) formats and conduct a comprehensive analysis to push their limits for efficient LLM serving. Our analysis shows that existing ultra low-bit BFP variants struggle to provide reasonable language model performance due to outlier values in blocks. To address the outliers with BFPs, we propose MX+, a cost-effective and non-intrusive extension designed for seamless integration into the microscaling (MX) formats. MX+ builds on the key insight that the outlier does not need to use its exponent field in the element data type, which allows us to repurpose the exponent field as an extended mantissa to increase the precision of the outlier element. Our evaluation shows that MX+ achieves significantly higher model performance compared to the 4-bit MX format (MXFP4) with negligible storage overhead and slowdown, thus offering a compelling alternative to MXFP4 or MXFP6 for efficient LLM inference.

Figures (14)

• Figure 1: Industry-driven block floating-point formats.
• Figure 2: Perplexity with BF16 baseline (B), MSFP, SMX, and MX formats. We compare each format with varying bit widths: high (H), moderate (M), and low (L).
• Figure 3: Perplexity of WikiText-2 across a mix of BF16 and MXFP4.
• Figure 4: (a) Heatmap of the sample attention input tensors of Llama-3.1-8B. (b) BF16 values and their MXFP4 and MXFP6 representations. BF16 values are rounded to the second decimal place for brevity.
• Figure 5: Contribution to MSE (%) from elements with the largest error in each MX block or from BM elements. We use the sampled attention input tensor of Layer 16.