Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast NF4 Dequantization Kernels for Large Language Model Inference

Xiangbo Qi, Chaoyi Jiang, Murali Annavaram

Abstract

Large language models (LLMs) have grown beyond the memory capacity of single GPU devices, necessitating quantization techniques for practical deployment. While NF4 (4-bit NormalFloat) quantization enables 4$\times$ memory reduction, inference on current NVIDIA GPUs (e.g., Ampere A100) requires expensive dequantization back to FP16 format, creating a critical performance bottleneck. This paper presents a lightweight shared memory optimization that addresses this gap through principled memory hierarchy exploitation while maintaining full ecosystem compatibility. We compare our technique against the open-source BitsAndBytes implementation, achieving 2.0--2.2$\times$ kernel speedup across three models (Gemma 27B, Qwen3 32B, and Llama3.3 70B) and up to 1.54$\times$ end-to-end improvement by leveraging the 12--15$\times$ latency advantage of shared memory over global memory access. Our optimization reduces instruction counts through simplified indexing logic while using only 64 bytes of shared memory per thread block, demonstrating that lightweight optimizations can deliver substantial performance gains with minimal engineering effort. This work provides a plug-and-play solution for the HuggingFace ecosystem that democratizes access to advanced models on existing GPU infrastructure.

Fast NF4 Dequantization Kernels for Large Language Model Inference

Abstract

Large language models (LLMs) have grown beyond the memory capacity of single GPU devices, necessitating quantization techniques for practical deployment. While NF4 (4-bit NormalFloat) quantization enables 4 memory reduction, inference on current NVIDIA GPUs (e.g., Ampere A100) requires expensive dequantization back to FP16 format, creating a critical performance bottleneck. This paper presents a lightweight shared memory optimization that addresses this gap through principled memory hierarchy exploitation while maintaining full ecosystem compatibility. We compare our technique against the open-source BitsAndBytes implementation, achieving 2.0--2.2 kernel speedup across three models (Gemma 27B, Qwen3 32B, and Llama3.3 70B) and up to 1.54 end-to-end improvement by leveraging the 12--15 latency advantage of shared memory over global memory access. Our optimization reduces instruction counts through simplified indexing logic while using only 64 bytes of shared memory per thread block, demonstrating that lightweight optimizations can deliver substantial performance gains with minimal engineering effort. This work provides a plug-and-play solution for the HuggingFace ecosystem that democratizes access to advanced models on existing GPU infrastructure.

Paper Structure

This paper contains 23 sections, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Model Inference
  4. NF4 Quantization Fundamentals
  5. Motivation
  6. The Quantization-Compute Gap
  7. Memory Access Pattern Analysis
  8. Profiling Analysis and Bottleneck Identification
  9. Problem Formulation and Design
  10. Optimization Objectives and Constraints
  11. Shared Memory Loading Strategy
  12. Memory Loading vs. Hardcoded Literals
  13. Simplified Index Computation
  14. Algorithm Design
  15. Thread-Level Architecture
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Baseline NF4 dequantization showing bottlenecks (1) 4-level tree decoding with branching overhead and warp divergence (2) Global memory code access at 290 clock cycles per operation
  • Figure 2: Memory-level architecture with shared NF4 LUT serving all lookups within the thread block.
  • Figure 3: Thread-level architecture with single-thread loading of LUT into shared memory.
  • Figure 4: End-to-end latency (top row) and throughput (bottom row) comparison across three models showing consistent improvements with our optimized NF4 dequantization kernel.