Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GWQ: Gradient-Aware Weight Quantization for Large Language Models

Yihua Shao, Yan Gu, Siyu Chen, Haiyang Liu, Zixian Zhu, Zijian Ling, Minxi Yan, Ziyang Yan, Chenyu Zhang, Michele Magno, Haotong Qin, Yan Wang, Jingcai Guo, Ling Shao, Hao Tang

TL;DR

GWQ introduces gradient-aware weight quantization for post-training compression of large language models, locating sensitive outliers via gradients from a single calibration sample and preserving the top 1% in FP16 while quantizing the rest to low-bit precision. The method performs per-channel, group-wise quantization with a gradient-based outlier mask, achieving data-efficient calibration and reduced inference memory. Across language modeling and multimodal tasks, GWQ delivers state-of-the-art or competitive results at average bit-width below 4 and yields about 1.2x faster inference compared to the original model. While leveraging gradient information offers robustness and data efficiency, it requires backpropagation and mixed-precision hardware considerations. Overall, GWQ provides a practical, gradient-guided PTQ paradigm that generalizes to both text-only and multimodal LLMs with meaningful performance and efficiency gains.

Abstract

Large language models (LLMs) show impressive performance in solving complex language tasks. However, its large number of parameters presents significant challenges for the deployment. So, compressing LLMs to low bits can enable to deploy on resource-constrained devices. To address this problem, we propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization that leverages gradients to localize outliers, requiring only a minimal amount of calibration data for outlier detection. GWQ retains the top 1\% outliers preferentially at FP16 precision, while the remaining non-outlier weights are stored in a low-bit. We widely evaluate GWQ on different task include language modeling, grounding detection, massive multitask language understanding and vision-language question and answering. Results show that models quantified by GWQ performs better than other quantization method. During quantization process, GWQ only need one calibration set to realize effective quant. Also, GWQ achieves 1.2x inference speedup in comparison to the original model and effectively reduces the inference memory.

GWQ: Gradient-Aware Weight Quantization for Large Language Models

TL;DR

GWQ introduces gradient-aware weight quantization for post-training compression of large language models, locating sensitive outliers via gradients from a single calibration sample and preserving the top 1% in FP16 while quantizing the rest to low-bit precision. The method performs per-channel, group-wise quantization with a gradient-based outlier mask, achieving data-efficient calibration and reduced inference memory. Across language modeling and multimodal tasks, GWQ delivers state-of-the-art or competitive results at average bit-width below 4 and yields about 1.2x faster inference compared to the original model. While leveraging gradient information offers robustness and data efficiency, it requires backpropagation and mixed-precision hardware considerations. Overall, GWQ provides a practical, gradient-guided PTQ paradigm that generalizes to both text-only and multimodal LLMs with meaningful performance and efficiency gains.

Abstract

Large language models (LLMs) show impressive performance in solving complex language tasks. However, its large number of parameters presents significant challenges for the deployment. So, compressing LLMs to low bits can enable to deploy on resource-constrained devices. To address this problem, we propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization that leverages gradients to localize outliers, requiring only a minimal amount of calibration data for outlier detection. GWQ retains the top 1\% outliers preferentially at FP16 precision, while the remaining non-outlier weights are stored in a low-bit. We widely evaluate GWQ on different task include language modeling, grounding detection, massive multitask language understanding and vision-language question and answering. Results show that models quantified by GWQ performs better than other quantization method. During quantization process, GWQ only need one calibration set to realize effective quant. Also, GWQ achieves 1.2x inference speedup in comparison to the original model and effectively reduces the inference memory.

Paper Structure

This paper contains 13 sections, 18 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Method
  4. Sensitive Weight Location
  5. Gradient Aware Quantization
  6. Experiments
  7. Experiments Setting
  8. Main Result
  9. Ablation Study
  10. Conclusion
  11. Limitation
  12. Appendix
  13. Next Token Prediction Accuracy

Figures (3)

  • Figure 1: Compared of GWQ and SPQR. Compared to SPQR, the outliers searched by GWQ are much sparser and easier to quantization.
  • Figure 2: Sensitive weight Location. GWQ utilizes a single calibration sample to calibrate and backpropagates this sample with the subsequent token from the large model's output as the target label. By halting the backpropagation process, GWQ is able to capture the model gradient $g$ prior to the update of the weights $W$. Within each layer, the gradients associated with each weight block are sorted in relation to their respective weights, and the top 1% with the largest $\left | g \right |$ are identified as the model's outliers, which are the weights that are particularly sensitive.
  • Figure 3: Performance of Different Quantized Qwen-VL. GWQ performs well compared to other quantization methods in the zero-shot grounding detection task.