AMS-QUANT: Adaptive Mantissa Sharing for Floating-point Quantization

TL;DR

The paper tackles the efficiency bottlenecks of large language models by introducing AMS-Quant, a non-integer floating-point quantization method. It combines adaptive Mantissa Sharing with Adaptive Searching to realize FP$(x-1).y$ quantization (e.g., FP5.3, FP4.25) while maintaining FP16-like accuracy, and it demonstrates substantial memory savings and wall-clock speedups via CUDA-optimized restoration and memory packing. Experimental results across multiple LLMs show FP5.3-e2m3 matches FP6-e2m3 in accuracy with higher compression, and FP4.25-e2m2 achieves near-best accuracy with pronounced speedups, identifying FP4.25 as a practical sweet spot. This work expands quantization to non-integer bit widths and provides actionable kernel and memory-layout strategies for memory-bound decoding tasks.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in various kinds of tasks, while the billion or even trillion parameters bring storage and efficiency bottlenecks for inference. Quantization, particularly floating-point quantization, is known to be capable of speeding up LLM inference by reducing memory footprint and data movement during the inference process. For the first time, we advance the floating-point quantization exploration from integer bitwidths to non-integer bit-widths, namely AMS-Quant, to further approach the quantization sweet spot. AMS-Quant incorporates two novel techniques to put it into effect: (1) it proposes Mantissa-bit Sharing, which groups k quantized weights and lets them share the least significant mantissa bit, allowing us to further approach the minimum quantization bit-width without accuracy loss. (2) It introduces Adaptive Searching, which employs an offline optimization strategy to minimize the accuracy degradation introduced by sharing. Moreover, AMS-Quant is also prototyped as efficient CUDA Linear kernels, which translates memory savings into wall-clock latency reduction by reducing memory access. Extensive experiments on large-scale datasets and models show that AMS-Quant can quantize the model to FP-5.33-e2m3 and FP4.25-e2m2, and significantly speed up the LLM decoding over FP16 inference (2.8x and 3.2x), with negligible accuracy loss.

Figures (6)

• Figure 1: Illustration of AMS-Quant Mantissa sharing. For the sake of simplicity, we take FP4.25 as an example.
• Figure 2: Figure \ref{['fig:fp-dist']} shows the value distribution of FP formats, and Figure \ref{['fig:weight-dist']} shows the weight distribution of 4 randomly chosen layers from different models. All follow bell-shaped distributions.
• Figure 3: Different RTN quantized models of Llama-3.2-3B-Instruct and Qwen3-4B-Instruct-2507 on GSM8k. FP6-e2m3 could retain the same accuracy level as the full precision of FP16, while FP6-e3m2 and FP5-e2m2 incur little performance degradation,and the FP4-e2m1 suffers from a serious loss.
• Figure 4: Illustrative example of restoring a prepacked AMS-Quant data back to FP16 via bit-level operations.
• Figure 5: Average scores of accuracy benchmark results for the 4 chosen models. We plotted the results in the order of decreasing bit-width to visualize the trend of accuracy changes(drops). For all the tested models, FP5.3 consistently shows the same accuracy level as FP16. The sharpest turning point appears at FP4.3 or FP4.25, indicating the quantization sweet spot.