Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview

Yanshu Wang, Tong Yang, Xiyan Liang, Guoan Wang, Hanning Lu, Xu Zhe, Yaoming Li, Li Weitao

TL;DR

This paper delve into various quantization techniques, including both post-training quantization (PTQ) and quantization-aware training (QAT), and analyze several state-of-the-art algorithms such as LLM-QAT, PEQA(L4Q), ZeroQuant, SmoothQuant, and others.

Abstract

This paper provides a comprehensive overview of the principles, challenges, and methodologies associated with quantizing large-scale neural network models. As neural networks have evolved towards larger and more complex architectures to address increasingly sophisticated tasks, the computational and energy costs have escalated significantly. We explore the necessity and impact of model size growth, highlighting the performance benefits as well as the computational challenges and environmental considerations. The core focus is on model quantization as a fundamental approach to mitigate these challenges by reducing model size and improving efficiency without substantially compromising accuracy. We delve into various quantization techniques, including both post-training quantization (PTQ) and quantization-aware training (QAT), and analyze several state-of-the-art algorithms such as LLM-QAT, PEQA(L4Q), ZeroQuant, SmoothQuant, and others. Through comparative analysis, we examine how these methods address issues like outliers, importance weighting, and activation quantization, ultimately contributing to more sustainable and accessible deployment of large-scale models.

Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview

TL;DR

This paper delve into various quantization techniques, including both post-training quantization (PTQ) and quantization-aware training (QAT), and analyze several state-of-the-art algorithms such as LLM-QAT, PEQA(L4Q), ZeroQuant, SmoothQuant, and others.

Abstract

This paper provides a comprehensive overview of the principles, challenges, and methodologies associated with quantizing large-scale neural network models. As neural networks have evolved towards larger and more complex architectures to address increasingly sophisticated tasks, the computational and energy costs have escalated significantly. We explore the necessity and impact of model size growth, highlighting the performance benefits as well as the computational challenges and environmental considerations. The core focus is on model quantization as a fundamental approach to mitigate these challenges by reducing model size and improving efficiency without substantially compromising accuracy. We delve into various quantization techniques, including both post-training quantization (PTQ) and quantization-aware training (QAT), and analyze several state-of-the-art algorithms such as LLM-QAT, PEQA(L4Q), ZeroQuant, SmoothQuant, and others. Through comparative analysis, we examine how these methods address issues like outliers, importance weighting, and activation quantization, ultimately contributing to more sustainable and accessible deployment of large-scale models.
Paper Structure (30 sections, 17 equations, 4 figures, 5 tables)

This paper contains 30 sections, 17 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Machine Learning
  4. Neural Networks the Evolution Towards Larger Models
  5. The Necessity and Impact of Model Size Growth
  6. Objectives, Importance, and Fundamental Methods of Model Quantization
  7. Fundamental Approaches
  8. LLM-QAT
  9. PEQA
  10. QLORA
  11. LUT-GEMM
  12. ZeroQuant
  13. SmoothQuant
  14. SpQR
  15. OliVe
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Comparison of Different Algorithms for Quantizing Weight Matrices. Some algorithms, such as RTN and LLM-QATliu2024spinquantllmquantizationlearned, directly quantize the weight matrix. Others, like SmoothQuantxiao2023smoothquant, SpQRdettmers2023spqr, OliVeguo2023olive, and EasyQuantTang2024EasyQuantAE, process outliers separately from normal values. Algorithms like GPTQfrantar2022gptq and QLoRadettmers2024qlora use matrix operation properties to preserve outliers during computations. Additionally, ZeroQuantyao2022zeroquant, SpQRdettmers2023spqr, and GPTQfrantar2022gptq address fine-grained quantization issues.
  • Figure 2: L4Q Comparison from jeon2024l4q
  • Figure 3: Summary of KV Cache Compression
  • Figure 4: Timeline of QAT and PTQ. The red highlighted methods represent they belonging to QAT-related methods, and others are PTQ-based methods.