Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models

TL;DR

This work provides the first systematic examination of quantizing reasoning-enabled LLMs, evaluating weight-only, KV-cache, and weight-activation quantization across a broad set of model families (Qwen, QwQ, LLaMA) and sizes (1.5B–70B). Through benchmarks like AIME-120, MATH-500, GSM8K, GPQA-Diamond, and LiveCodeBench, the authors compare several quantization algorithms (AWQ, GPTQ, QuaRot, FlatQuant, SmoothQuant, MXFP4) and establish practical lossless configurations, notably 8-bit weight-activation and 4-bit quantization for weights or KV cache depending on size. The study reveals that model origin and task difficulty critically shape quantization robustness, with larger models generally tolerating quantization better and harder tasks suffering greater degradation, while calibration data domain can strongly influence GPTQ performance. Importantly, they show that quantized models do not universally produce longer outputs, and that test-time scaling and careful model-size selection can improve accuracy-efficiency trade-offs. All results and code are open-sourced, providing actionable guidance for deploying efficient reasoning systems.

Abstract

Recent advancements in reasoning language models have demonstrated remarkable performance in complex tasks, but their extended chain-of-thought reasoning process increases inference overhead. While quantization has been widely adopted to reduce the inference cost of large language models, its impact on reasoning models remains understudied. In this paper, we conduct the first systematic study on quantized reasoning models, evaluating the open-sourced DeepSeek-R1-Distilled Qwen and LLaMA families ranging from 1.5B to 70B parameters, QwQ-32B, and Qwen3-8B. Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths, with extensive evaluation across mathematical (AIME, MATH-500), scientific (GPQA), and programming (LiveCodeBench) reasoning benchmarks. Our findings reveal that while lossless quantization can be achieved with W8A8 or W4A16 quantization, lower bit-widths introduce significant accuracy risks. We further identify model size, model origin, and task difficulty as critical determinants of performance. Contrary to expectations, quantized models do not exhibit increased output lengths. In addition, strategically scaling the model sizes or reasoning steps can effectively enhance the performance. All quantized models and codes are open-sourced in https://github.com/ruikangliu/Quantized-Reasoning-Models.

Figures (11)

• Figure 1: Performance of the quantized DeepSeek-R1-Distill-Qwen models on five benchmarks: AIME-120, MATH-500, GSM8K, GPQA-Diamond, and LiveCodeBench. Each chart presents the performance of different quantization strategies compared to the BF16 baseline.
• Figure 2: (a)-(c) shows the comparisons between DeepSeek-R1-Distill-Qwen-32B (SFT-based) and QwQ-32B (RL-based) on different quantization configurations. (d)-(f) shows comparisons between DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-LLaMA-8B on different quantization configurations.
• Figure 3: The lengths of generated tokens across different datasets and models.
• Figure 4: Model size and average latency v.s. accuracy (%) on LiveCodeBench.
• Figure 5: Average reasoning tokens v.s. accuracy (%) on DS-R1-Distill-Qwen models.