Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resource-Efficient Language Models: Quantization for Fast and Accessible Inference

Tollef Emil Jørgensen

TL;DR

This work addresses the resource and latency challenges of large language models by focusing on post-training quantization (PTQ) as a practical pathway for end-user inference on standard hardware. It provides a comprehensive, taxonomy-driven overview of PTQ methods, including asymmetric and symmetric quantization, static and dynamic quantization, and granularity choices (per tensor, per channel, per group), along with parameter-selection strategies such as min-max, percentile, MSE, and cross-entropy. The paper catalogs core PTQ methods and libraries (e.g., ZeroQuant, GPTQ, AWQ, SmoothQuant, HQQ) and discusses trade-offs between accuracy, speed, and hardware compatibility, while highlighting ongoing challenges like activation outliers and data-free calibration. Collectively, the work offers practical guidance for deploying quantized LLMs on consumer hardware, underlining automated calibration and case-specific evaluation as promising directions for making large models broadly accessible and energy-efficient.

Abstract

Large language models have significantly advanced natural language processing, yet their heavy resource demands pose severe challenges regarding hardware accessibility and energy consumption. This paper presents a focused and high-level review of post-training quantization (PTQ) techniques designed to optimize the inference efficiency of LLMs by the end-user, including details on various quantization schemes, granularities, and trade-offs. The aim is to provide a balanced overview between the theory and applications of post-training quantization.

Resource-Efficient Language Models: Quantization for Fast and Accessible Inference

TL;DR

This work addresses the resource and latency challenges of large language models by focusing on post-training quantization (PTQ) as a practical pathway for end-user inference on standard hardware. It provides a comprehensive, taxonomy-driven overview of PTQ methods, including asymmetric and symmetric quantization, static and dynamic quantization, and granularity choices (per tensor, per channel, per group), along with parameter-selection strategies such as min-max, percentile, MSE, and cross-entropy. The paper catalogs core PTQ methods and libraries (e.g., ZeroQuant, GPTQ, AWQ, SmoothQuant, HQQ) and discusses trade-offs between accuracy, speed, and hardware compatibility, while highlighting ongoing challenges like activation outliers and data-free calibration. Collectively, the work offers practical guidance for deploying quantized LLMs on consumer hardware, underlining automated calibration and case-specific evaluation as promising directions for making large models broadly accessible and energy-efficient.

Abstract

Large language models have significantly advanced natural language processing, yet their heavy resource demands pose severe challenges regarding hardware accessibility and energy consumption. This paper presents a focused and high-level review of post-training quantization (PTQ) techniques designed to optimize the inference efficiency of LLMs by the end-user, including details on various quantization schemes, granularities, and trade-offs. The aim is to provide a balanced overview between the theory and applications of post-training quantization.
Paper Structure (45 sections, 12 equations, 10 figures, 1 table)

This paper contains 45 sections, 12 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. In-depth Background
  5. QAT
  6. PTQ
  7. No "One size fits all" Solution
  8. Assymetric Quantization
  9. Asym. Quantization Function
  10. Asym. Dequantization Function
  11. Symmetric Quantization
  12. Sym. Quantization Function
  13. Sym. Dequantization Function
  14. Static Quantization
  15. Dynamic Quantization
  16. ...and 30 more sections

Figures (10)

  • Figure 1: Quantization and dequantization of a $3\times3$ matrix visualized, with rounded values for readability. In the process from FP32 to INT8 and back to FP32, most values will deviate.
  • Figure 2: High-magnitude activation outliers and value distribution in the MLP output projection of layers 1, 4, 8, and 12 after a forward pass with the sentence "model quantization is awesome", using the 125M-parameter model GPT-Neo gpt-neo.
  • Figure 3: Workflow of quantization-aware training.
  • Figure 4: Workflow of post-training quantization.
  • Figure 5: Quantization granularities -- per tensor, per channel (row or column), or by determined group sizes.
  • ...and 5 more figures