Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization

Ji-Fu Li, Manyi Zhang, Xiaobo Xia, Han Bao, Haoli Bai, Zhenhua Dong, Xianzhi Yu

Abstract

Microscaling floating-point (MXFP) formats have emerged as a promising standard for deploying Multi-modal Large Language Models (MLLMs) and Large Language Models (LLMs) on modern accelerator architectures. However, existing Post-Training Quantization (PTQ) methods, particularly rotation-based techniques designed for integer formats, suffer from severe performance collapse when applied to MXFP4. Recent studies attribute this failure to a fundamental format mismatch: global orthogonal rotations inadvertently transfer outlier energy across quantization blocks, inducing new outliers that disrupt local block-wise scaling, while often creating bimodal activation distributions that underutilize the limited quantization range. To address these issues, we propose BATQuant (Block-wise Affine Transformation), which restricts transformations to align with MXFP granularity to prevent cross-block outlier propagation, while relaxing orthogonality constraints to optimize distribution shaping. To ensure parameter efficiency, we introduce Global and Private Kronecker (GPK) decomposition to effectively reduces storage and runtime overhead and incorporate Block-wise Learnable Clipping to suppress residual outliers. Extensive experiments on both MLLMs and LLMs demonstrate that BATQuant establishes new state-of-the-art results under aggressive W4A4KV16 configurations, recovering up to 96.43% of full-precision performance on multimodal benchmarks and clearly outperforming existing methods across diverse tasks.

BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization

Abstract

Microscaling floating-point (MXFP) formats have emerged as a promising standard for deploying Multi-modal Large Language Models (MLLMs) and Large Language Models (LLMs) on modern accelerator architectures. However, existing Post-Training Quantization (PTQ) methods, particularly rotation-based techniques designed for integer formats, suffer from severe performance collapse when applied to MXFP4. Recent studies attribute this failure to a fundamental format mismatch: global orthogonal rotations inadvertently transfer outlier energy across quantization blocks, inducing new outliers that disrupt local block-wise scaling, while often creating bimodal activation distributions that underutilize the limited quantization range. To address these issues, we propose BATQuant (Block-wise Affine Transformation), which restricts transformations to align with MXFP granularity to prevent cross-block outlier propagation, while relaxing orthogonality constraints to optimize distribution shaping. To ensure parameter efficiency, we introduce Global and Private Kronecker (GPK) decomposition to effectively reduces storage and runtime overhead and incorporate Block-wise Learnable Clipping to suppress residual outliers. Extensive experiments on both MLLMs and LLMs demonstrate that BATQuant establishes new state-of-the-art results under aggressive W4A4KV16 configurations, recovering up to 96.43% of full-precision performance on multimodal benchmarks and clearly outperforming existing methods across diverse tasks.
Paper Structure (40 sections, 6 equations, 13 figures, 7 tables, 2 algorithms)

This paper contains 40 sections, 6 equations, 13 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Method
  4. Block-wise Affine Transformation
  5. Global and Private Kronecker.
  6. Block-wise Learnable Clipping.
  7. The Training Objective.
  8. Integration with the Transformer Architecture
  9. MLP Module.
  10. Self-Attention Module.
  11. Experiments
  12. Settings
  13. Evaluation and Baselines.
  14. Implementation Details.
  15. Quantization Settings.
  16. ...and 25 more sections

Figures (13)

  • Figure 1: Quantization performance on Qwen3-VL-8B-Instruct across various methods. Our method yields superior results compared to baselines across all bit-width settings. The advantage is particularly substantial in the W4A4 setting, where our method clearly outperforms existing methods.
  • Figure 2: Activation distributions for the down_proj module in layer 35 of Qwen3-8B. The central 3D plots illustrate the activations after transformation. We specifically extract Block 5 (without outliers) and Block 295 (with extreme outliers), and visualize the values after scaling factor division but prior to rounding. (a) After applying the block Hadamard transform, block 295 exhibits a bimodal distribution, leading to inefficient utilization of the bit width. (b) After the block affine transformation, block 295 shows reduced magnitude compared to subplot (a) while effectively leveraging the floating-point quantization grids.
  • Figure 3: The overall framework of BATQuant.Bottom: Integration of BATQuant into the Transformer architecture. Weight-side transformations are fused offline into the linear layers, while activation-side transformations are applied online. Top: Exemplary view of the Block-wise Affine Transformation, where inputs are partitioned into MXFP-aligned blocks. Each block transformation is decomposed via the Global and Private Kronecker.
  • Figure 4: Performance comparison of different methods on Qwen3-8B across LLM benchmarks under various quantization configurations. The results are categorized into Non-Reasoning (left) and Reasoning (right) tasks.
  • Figure 5: Activation distributions of the q_proj module in layer 6 of Qwen3-8B with different quantization methods.
  • ...and 8 more figures