Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Study on Quantization Techniques for Large Language Models

Jiedong Lang, Zhehao Guo, Shuyu Huang

TL;DR

This research provides a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs.

Abstract

Large Language Models (LLMs) have been extensively researched and used in both academia and industry since the rise in popularity of the Transformer model, which demonstrates excellent performance in AI. However, the computational demands of LLMs are immense, and the energy resources required to run them are often limited. For instance, popular models like GPT-3, with 175 billion parameters and a storage requirement of 350 GB, present significant challenges for deployment on resource-constrained IoT devices and embedded systems. These systems often lack the computational capacity to handle such large models. Quantization, a technique that reduces the precision of model values to a smaller set of discrete values, offers a promising solution by reducing the size of LLMs and accelerating inference. In this research, we provide a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs. We begin by exploring the mathematical theory of quantization, followed by a review of common quantization methods and how they are implemented. Furthermore, we examine several prominent quantization methods applied to LLMs, detailing their algorithms and performance outcomes.

A Comprehensive Study on Quantization Techniques for Large Language Models

TL;DR

This research provides a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs.

Abstract

Large Language Models (LLMs) have been extensively researched and used in both academia and industry since the rise in popularity of the Transformer model, which demonstrates excellent performance in AI. However, the computational demands of LLMs are immense, and the energy resources required to run them are often limited. For instance, popular models like GPT-3, with 175 billion parameters and a storage requirement of 350 GB, present significant challenges for deployment on resource-constrained IoT devices and embedded systems. These systems often lack the computational capacity to handle such large models. Quantization, a technique that reduces the precision of model values to a smaller set of discrete values, offers a promising solution by reducing the size of LLMs and accelerating inference. In this research, we provide a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs. We begin by exploring the mathematical theory of quantization, followed by a review of common quantization methods and how they are implemented. Furthermore, we examine several prominent quantization methods applied to LLMs, detailing their algorithms and performance outcomes.

Paper Structure

This paper contains 13 sections, 7 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. RANGE MAPPING
  3. AFFINE QUATIZATION
  4. THE EQUATION BEHIND THE AFFINE QUATIZATION
  5. THE PARAMETERS BEHIND THE EQUATION
  6. SCALE QUATIZATION
  7. THE EQUATION BEHIND THE SCALE QUATIZATION
  8. DETAILS OF SCALE QUANTIZATION
  9. QUANTIZATION GRANULARITY
  10. QUANTIZATION CALIBRATION
  11. QUANTIZATION TECHNIQUES
  12. Analysis of Quantization Approaches in LLMs
  13. Conclusion

Figures (2)

  • Figure 1: GPTQ quantizes the weights in blocks, with each block consisting of 128 columns, similar to a sliding window approach. This block-wise quantization allows the model to process smaller segments of weights at a time, improving computational efficiency while maintaining accuracy during the quantization process.
  • Figure 2: