Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning

Zhen Li, Yupeng Su, Songmiao Wang, Runming Yang, Congkai Xie, Aofan Liu, Ming Li, Jiannong Cao, Ngai Wong, Hongxia Yang

TL;DR

This work investigates how low-$4$-bit post-training quantization degrades mathematical reasoning in open-source LLMs and unveils that degradation predominantly affects procedural and executional steps, often cascading from the first faulty step. It introduces a measure$ ightarrow$locate$ ightarrow$restore pipeline that uses targeted, small-scale supervision via compact Silver Bullet datasets and Direct Preference Optimization to recover reasoning performance on quantized models, achieving near full-precision baselines with minimal data and compute. The authors demonstrate quantizer- and architecture-agnostic applicability across AWQ, GPTQ, and SmoothQuant on Qwen and LLaMA families over GSM8K, MATH, and AIME, and they validate the approach with extensive error taxonomy analysis, ablations, and restoration experiments. The practical impact lies in enabling efficient, robust deployment of reasoning-capable LLMs in resource-constrained environments without sacrificing core reasoning capabilities. The study also contributes a rigorous, automated error analysis framework and a scalable data-centric intervention that can generalize to other domains requiring precise stepwise reasoning.

Abstract

Low-bit post-training quantization (PTQ) is a practical route to deploy reasoning-capable LLMs under tight memory and latency budgets, yet it can markedly impair mathematical reasoning (drops up to 69.81% in our harder settings). We address two deployment-critical questions with process-level precision: Where along a step-structured solution does degradation first arise? How to mitigate it while staying in the low-bit regime? Across widely used PTQ methods (AWQ, GPTQ, SmoothQuant), open-source model families (Qwen, LLaMA; 0.5--7B), and math reasoning benchmarks (GSM8K, MATH, AIME), we perform format-aligned chain-of-thought with step-aligned attribution and uncover two robust regularities: (i) PTQ disproportionately elevates method and execution errors relative to high-level conceptual mistakes; and (ii) failures emerge early, with the first vulnerable step flipping and cascading to the final answer. These regularities suggest a general intervention principle: restore local token-level margins exactly at the earliest failure frontier. We instantiate this principle as a lightweight measure$\rightarrow$locate$\rightarrow$restore loop that operates directly on the quantized model: detect the first faulty step, construct our "Silver Bullet" datasets, and apply small-scale supervised/preference tuning. In our settings, as few as 332 curated examples and 3--5 minutes of compute on a single GPU recover 4-bit weight math reasoning toward the full-precision baseline while preserving PTQ efficiency. Our framework is quantizer- and architecture-agnostic within the evaluated regimes, and turns low-bit degradation from a global accuracy problem into a local, reproducible process intervention.

Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning

TL;DR

This work investigates how low--bit post-training quantization degrades mathematical reasoning in open-source LLMs and unveils that degradation predominantly affects procedural and executional steps, often cascading from the first faulty step. It introduces a measurelocaterestore pipeline that uses targeted, small-scale supervision via compact Silver Bullet datasets and Direct Preference Optimization to recover reasoning performance on quantized models, achieving near full-precision baselines with minimal data and compute. The authors demonstrate quantizer- and architecture-agnostic applicability across AWQ, GPTQ, and SmoothQuant on Qwen and LLaMA families over GSM8K, MATH, and AIME, and they validate the approach with extensive error taxonomy analysis, ablations, and restoration experiments. The practical impact lies in enabling efficient, robust deployment of reasoning-capable LLMs in resource-constrained environments without sacrificing core reasoning capabilities. The study also contributes a rigorous, automated error analysis framework and a scalable data-centric intervention that can generalize to other domains requiring precise stepwise reasoning.

Abstract

Low-bit post-training quantization (PTQ) is a practical route to deploy reasoning-capable LLMs under tight memory and latency budgets, yet it can markedly impair mathematical reasoning (drops up to 69.81% in our harder settings). We address two deployment-critical questions with process-level precision: Where along a step-structured solution does degradation first arise? How to mitigate it while staying in the low-bit regime? Across widely used PTQ methods (AWQ, GPTQ, SmoothQuant), open-source model families (Qwen, LLaMA; 0.5--7B), and math reasoning benchmarks (GSM8K, MATH, AIME), we perform format-aligned chain-of-thought with step-aligned attribution and uncover two robust regularities: (i) PTQ disproportionately elevates method and execution errors relative to high-level conceptual mistakes; and (ii) failures emerge early, with the first vulnerable step flipping and cascading to the final answer. These regularities suggest a general intervention principle: restore local token-level margins exactly at the earliest failure frontier. We instantiate this principle as a lightweight measurelocaterestore loop that operates directly on the quantized model: detect the first faulty step, construct our "Silver Bullet" datasets, and apply small-scale supervised/preference tuning. In our settings, as few as 332 curated examples and 3--5 minutes of compute on a single GPU recover 4-bit weight math reasoning toward the full-precision baseline while preserving PTQ efficiency. Our framework is quantizer- and architecture-agnostic within the evaluated regimes, and turns low-bit degradation from a global accuracy problem into a local, reproducible process intervention.
Paper Structure (42 sections, 5 equations, 3 figures, 4 tables)

This paper contains 42 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Quantization Methods
  4. Reasoning Ability Optimization in Large Language Models
  5. Methodology
  6. Quantization-Induced Degradation: Measurement and Attribution
  7. Quantization
  8. Format Alignment Training
  9. Detailed Examination of Reasoning Process
  10. Restoring Reasoning Abilities in Quantized Models
  11. Data Extraction
  12. Silver Bullet Datasets Building
  13. Capability Restoring Training
  14. Experiments
  15. Evaluating Quantization Effects
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Pipeline of our study for investigating and restoring mathematical reasoning capabilities in quantized language models. We begin by identifying performance degradation caused by quantization, then apply format alignment training and a structured error assessment pipeline involving expert model judgments. Through this process, we analyze reasoning failures in step-by-step outputs. Targeted "Silver Bullet" datasets are constructed based on consensus error types, and used in DPO training to recover reasoning performance while maintaining the efficiency of low-bit models.
  • Figure 2: Error assessment results for full‐precision and quantized models. For the full‐precision model, we aggregate all problems it answered incorrectly; for each quantized model, we count only those problems that the full‐precision model solved correctly but the quantized model failed, enabling comparison of quantization‐induced changes across error dimensions.
  • Figure 3: Relative capability restoration with our method. Radar values are normalized to each model’s Vanilla-FP16 accuracy on the same benchmark (radius 1.0). Solid = After Restoration, dashed = Before Restoration (AWQ, GPTQ).