FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference

TL;DR

FGMP introduces a post-training, fine-grained mixed-precision quantization framework for LLM inference that preserves accuracy by keeping a small fraction of blocks in higher precision (FP8) while quantizing the majority to FP4, guided by a Fisher-information-based sensitivity policy. The method couples a global, offline thresholding strategy with a sensitivity-weighted clipping technique and is implemented in hardware via a block-granularity VMAC datapath and a mixed-precision activation quantization unit for on-the-fly decisions. Empirical results on Llama-2-7B with Wikitext-103 show less than 1% perplexity degradation at competitive compression, along with up to 39% memory savings and about 14% energy reductions, while preserving downstream task performance. This work demonstrates a practical, hardware-aware path to efficient LLM inference using predominantly low-precision compute without sacrificing accuracy.

Abstract

Quantization is a powerful tool to improve large language model (LLM) inference efficiency by utilizing more energy-efficient low-precision datapaths and reducing memory footprint. However, accurately quantizing LLM weights and activations to low precision is challenging without degrading model accuracy. We propose fine-grained mixed precision (FGMP) quantization, a post-training mixed-precision quantization hardware-software co-design methodology that maintains accuracy while quantizing the majority of weights and activations to reduced precision. Our work makes the following contributions: 1) We develop a policy that uses the perturbation in each value, weighted by the Fisher information, to select which weight and activation blocks to keep in higher precision. This approach preserves accuracy by identifying which weight and activation blocks need to be retained in higher precision to minimize the perturbation in the model loss. 2) We also propose a sensitivity-weighted clipping approach for fine-grained quantization which helps retain accuracy for blocks that are quantized to low precision. 3) We then propose hardware augmentations to leverage the efficiency benefits of FGMP quantization. Our hardware implementation encompasses i) datapath support for FGMP at block granularity, and ii) a mixed-precision activation quantization unit to assign activation blocks to high or low precision on the fly with minimal runtime and energy overhead. Our design, prototyped using NVFP4 (an FP4 format with microscaling) as the low-precision datatype and FP8 as the high-precision datatype, facilitates efficient FGMP quantization, attaining <1% perplexity degradation on Wikitext-103 for the Llama-2-7B model relative to an all-FP8 baseline design while consuming 14% less energy during inference and requiring 30% less weight memory.

Figures (10)

• Figure 1: Perplexity degradation (less is better) versus compression rate (higher is better), evaluated using Llama-2-7B on Wikitext-103 with 4-bit weight-activation quantization. Compression rate is computed as 16 divided by the average bit width for weights and activations (assuming 4K context length). We group prior work into algorithm-only approaches that use integer quantization ("Algo."), prior work on microscaling quantization ("$\mu$scale"), and prior methods for mixed-precision quantization ("MP"). Our method, FGMP, (shown here with 70%, 80%, and 90% of blocks in FP4) attains reduced perplexity degradation relative to existing post-training quantization methods.
• Figure 2: a) Diagram outlining our approach for FGMP quantization at block granularity. We quantize both weights and activations to mixed precision, preserving sensitive blocks in higher precision (FP8) and quantizing the rest to low precision (NVFP4). b) Visualization of a slice of the activation and weight precision assignments for Layer 7 Fully Connected 1 ("FC1") in the Llama-2-7B model using our policy with 10% of blocks in FP8 and with our sensitivity-weighted clipping method applied. The NVFP4 and FP8 blocks are interleaved in an unstructured manner.
• Figure 3: a) High-level accelerator architecture, consisting of a PE array (each of which contains a VMAC-based datapath), as well as one or more vector units which contain our post-processing activation quantization unit (outlined in detail in Figure \ref{['fig:main-hw2']}). b) Datapath support for FGMP quantization with four dot-product units per lane to perform FGMP VMAC operations. The portion of the figure highlighted in gray is synthesized for hardware measurements.
• Figure 4: A diagram of the post-processing unit for quantizing activations to mixed-precision online during inference.
• Figure 5: Perplexity evaluation on Wikitext-103 for different models in the Llama-2, GPT3, and Nemotron model families. We report FGMP quantization results with and without our sensitivity-weighted weight clipping approach. Note that for GPT3-22B and Nemotron-4-15B, we did not apply sensitivity-weighted clipping as it did not yield perplexity improvements, and for Nemotron-4-15B, we exclude the BF16 baseline as it exhibited worse perplexity than FP8.