Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study of Qwen3 Quantization

Xingyu Zheng, Yuye Li, Haoran Chu, Yue Feng, Xudong Ma, Jie Luo, Jinyang Guo, Haotong Qin, Michele Magno, Xianglong Liu

TL;DR

This study systematically evaluates the robustness of Qwen3 under post-training quantization across five classic PTQ methods and bit-widths ranging from 1 to 8, using a controlled calibration protocol and a diverse set of benchmarks including perplexity and zero-/few-shot reasoning tasks. It reveals near lossless performance at 8-bit weight quantization but substantial degradation at 3-bit and below, with activation quantization (e.g., SmoothQuant) often causing greater harm than weight-only quantization; larger models tend to be more stable than smaller ones, yet Qwen3 remains more sensitive to ultra-low-bit settings than LLaMA3 in comparable configurations. The findings highlight the need for novel quantization techniques tailored to modern, heavily pre-trained LLMs and provide practical guidance for deploying these models in resource-constrained environments. The work also outlines future directions in channel reordering and rotation-based quantization to better preserve accuracy at ultra-low bit-widths.

Abstract

The Qwen series has emerged as a leading family of open-source Large Language Models (LLMs), demonstrating remarkable capabilities in natural language understanding tasks. With the recent release of Qwen3, which exhibits superior performance across diverse benchmarks, there is growing interest in deploying these models efficiently in resource-constrained environments. Low-bit quantization presents a promising solution, yet its impact on Qwen3's performance remains underexplored. This study conducts a systematic evaluation of Qwen3's robustness under various quantization settings, aiming to uncover both opportunities and challenges in compressing this state-of-the-art model. We rigorously assess 5 existing classic post-training quantization techniques applied to Qwen3, spanning bit-widths from 1 to 8 bits, and evaluate their effectiveness across multiple datasets. Our findings reveal that while Qwen3 maintains competitive performance at moderate bit-widths, it experiences notable degradation in linguistic tasks under ultra-low precision, underscoring the persistent hurdles in LLM compression. These results emphasize the need for further research to mitigate performance loss in extreme quantization scenarios. We anticipate that this empirical analysis will provide actionable insights for advancing quantization methods tailored to Qwen3 and future LLMs, ultimately enhancing their practicality without compromising accuracy. Our project is released on https://github.com/Efficient-ML/Qwen3-Quantization and https://huggingface.co/collections/Efficient-ML/qwen3-quantization-68164450decb1c868788cb2b.

An Empirical Study of Qwen3 Quantization

TL;DR

This study systematically evaluates the robustness of Qwen3 under post-training quantization across five classic PTQ methods and bit-widths ranging from 1 to 8, using a controlled calibration protocol and a diverse set of benchmarks including perplexity and zero-/few-shot reasoning tasks. It reveals near lossless performance at 8-bit weight quantization but substantial degradation at 3-bit and below, with activation quantization (e.g., SmoothQuant) often causing greater harm than weight-only quantization; larger models tend to be more stable than smaller ones, yet Qwen3 remains more sensitive to ultra-low-bit settings than LLaMA3 in comparable configurations. The findings highlight the need for novel quantization techniques tailored to modern, heavily pre-trained LLMs and provide practical guidance for deploying these models in resource-constrained environments. The work also outlines future directions in channel reordering and rotation-based quantization to better preserve accuracy at ultra-low bit-widths.

Abstract

The Qwen series has emerged as a leading family of open-source Large Language Models (LLMs), demonstrating remarkable capabilities in natural language understanding tasks. With the recent release of Qwen3, which exhibits superior performance across diverse benchmarks, there is growing interest in deploying these models efficiently in resource-constrained environments. Low-bit quantization presents a promising solution, yet its impact on Qwen3's performance remains underexplored. This study conducts a systematic evaluation of Qwen3's robustness under various quantization settings, aiming to uncover both opportunities and challenges in compressing this state-of-the-art model. We rigorously assess 5 existing classic post-training quantization techniques applied to Qwen3, spanning bit-widths from 1 to 8 bits, and evaluate their effectiveness across multiple datasets. Our findings reveal that while Qwen3 maintains competitive performance at moderate bit-widths, it experiences notable degradation in linguistic tasks under ultra-low precision, underscoring the persistent hurdles in LLM compression. These results emphasize the need for further research to mitigate performance loss in extreme quantization scenarios. We anticipate that this empirical analysis will provide actionable insights for advancing quantization methods tailored to Qwen3 and future LLMs, ultimately enhancing their practicality without compromising accuracy. Our project is released on https://github.com/Efficient-ML/Qwen3-Quantization and https://huggingface.co/collections/Efficient-ML/qwen3-quantization-68164450decb1c868788cb2b.
Paper Structure (5 sections, 4 figures, 4 tables)

This paper contains 5 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Empirical Study
  3. Experiment Settings
  4. PTQ results
  5. Conclusion

Figures (4)

  • Figure 1: PPL of per-group quantization on C4 of Qwen3-Base.
  • Figure 2: 0-shot Commonsense Reasoning Accuracy (Average of PIQA/Arc-Easy/Arc-Challenge/HellaSwag/Winogrande/BoolQ) of per-group quantization of Qwen3-Base.
  • Figure 3: PPL of per-group (AWQ, GPTQ, BiLLM) and per-channel (SmoothQuant) quantization methods on c4 of Qwen3.
  • Figure 4: 0-shot Commonsense Reasoning Accuracy (Average of PIQA/Arc-Easy/Arc-Challenge/HellaSwag/Winogrande/BoolQ) per-group (AWQ, GPTQ, BiLLM) and per-channel (SmoothQuant) quantization methods on c4 of Qwen3.