Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats

Xiaoxia Wu, Zhewei Yao, Yuxiong He

TL;DR

The paper tackles the challenge of deploying large language models with low-precision quantization by evaluating floating-point PTQ using FP8 and FP4 formats. It demonstrates that FP8 activation outperforms INT8, FP8/FP4 weights are competitive or superior to their integer counterparts, and that Low Rank Compensation (LoRC) can significantly improve W4A8 accuracy, especially in smaller models. The authors introduce practical scaling constraints and casting techniques to align FP4 with FP8, achieving hardware-friendly efficiency with minimal performance loss. These findings make FP quantization a promising route for efficient LLM deployment on FP-enabled hardware like NVIDIA H100, enabling better performance in resource-constrained settings.

Abstract

In the complex domain of large language models (LLMs), striking a balance between computational efficiency and maintaining model quality is a formidable challenge. Navigating the inherent limitations of uniform quantization, particularly when dealing with outliers, and motivated by the launch of NVIDIA's H100 hardware, this study delves into the viability of floating-point (FP) quantization, particularly focusing on FP8 and FP4, as a potential solution. Our comprehensive investigation reveals that for LLMs, FP8 activation consistently outshines its integer (INT8) equivalent, with the performance edge becoming more noticeable in models possessing parameters beyond one billion. For weight quantization, our findings indicate that FP4 exhibits comparable, if not superior, performance to INT4, simplifying deployment on FP-supported hardware like H100. To mitigate the overhead from precision alignment caused by the disparity between weights and activations, we propose two scaling constraints for weight quantization that negligibly impact the performance compared to the standard W4A8 model. We additionally enhance our quantization methods by integrating the Low Rank Compensation (LoRC) strategy, yielding improvements especially in smaller models. The results of our investigation emphasize the immense potential of FP quantization for LLMs, paving the way for high-efficiency deployment in resource-limited settings.

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats

TL;DR

The paper tackles the challenge of deploying large language models with low-precision quantization by evaluating floating-point PTQ using FP8 and FP4 formats. It demonstrates that FP8 activation outperforms INT8, FP8/FP4 weights are competitive or superior to their integer counterparts, and that Low Rank Compensation (LoRC) can significantly improve W4A8 accuracy, especially in smaller models. The authors introduce practical scaling constraints and casting techniques to align FP4 with FP8, achieving hardware-friendly efficiency with minimal performance loss. These findings make FP quantization a promising route for efficient LLM deployment on FP-enabled hardware like NVIDIA H100, enabling better performance in resource-constrained settings.

Abstract

In the complex domain of large language models (LLMs), striking a balance between computational efficiency and maintaining model quality is a formidable challenge. Navigating the inherent limitations of uniform quantization, particularly when dealing with outliers, and motivated by the launch of NVIDIA's H100 hardware, this study delves into the viability of floating-point (FP) quantization, particularly focusing on FP8 and FP4, as a potential solution. Our comprehensive investigation reveals that for LLMs, FP8 activation consistently outshines its integer (INT8) equivalent, with the performance edge becoming more noticeable in models possessing parameters beyond one billion. For weight quantization, our findings indicate that FP4 exhibits comparable, if not superior, performance to INT4, simplifying deployment on FP-supported hardware like H100. To mitigate the overhead from precision alignment caused by the disparity between weights and activations, we propose two scaling constraints for weight quantization that negligibly impact the performance compared to the standard W4A8 model. We additionally enhance our quantization methods by integrating the Low Rank Compensation (LoRC) strategy, yielding improvements especially in smaller models. The results of our investigation emphasize the immense potential of FP quantization for LLMs, paving the way for high-efficiency deployment in resource-limited settings.
Paper Structure (6 sections, 1 equation, 2 figures, 4 tables)

This paper contains 6 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Main Results
  5. Conclusions
  6. Experiment Details

Figures (2)

  • Figure 1: Distribution of Activation values. The top, middle and bottom rows represents the distributions at the 2nd, 12th and final layer of the pretrained OPT-1.3b model. From the left to right columns, they are respectively for the linear modules attn.q_proj (same as attn.k_proj and attn.v_proj), attn.out_proj, fc1, and fc2. The histogram's x-axis ranges from the smallest to largest activation values, while the y-axis denotes their frequency in the dataset. See legend for their minimum and maximum values. Density functions illustrate the probability of different activation values. For more details, please see Section \ref{['sec:activation']}.
  • Figure 2: A Contrast between INT8 and FP8 Quantization Methods. The top row displays the original vector in its full-precision form. The subsequent row showcases the vector after quantization through the INT8 Asymmetric approach. The final two rows present values quantized by the FP8 method, utilizing E5M2 and E4M3 formats respectively.