D$^2$Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs

TL;DR

D$^2$Quant tackles sub-4-bit weight-only PTQ for LLMs by addressing two bottlenecks: down-projection fidelity and activation drift. It introduces a Dual-Scale Quantizer (DSQ) to inject a column-wise scale into down-projection quantization and a Deviation-Aware Correction (DAC) that uses a signal-to-noise ratio (SNR) based bias alignment at post-attention LayerNorm, with $SNR_i = \mu_i^2/\sigma_i^2$ and an error-reduction bound of $\frac{\mathrm{SNR}_i}{1+\mathrm{SNR}_i}$. The approach yields a unified, block-wise PTQ pipeline that significantly improves perplexity and downstream task accuracy across Qwen and LLaMA families at 2-bit quantization, outperforming GPTQ, GPTAQ, and BoA with minimal overhead. By keeping weight and activation corrections lightweight and absorbable, D$^2$Quant enables practical, deployment-friendly low-bit quantization for diverse LLMs.

Abstract

Large language models (LLMs) deliver strong performance, but their high compute and memory costs make deployment difficult in resource-constrained scenarios. Weight-only post-training quantization (PTQ) is appealing, as it reduces memory usage and enables practical speedup without low-bit operators or specialized hardware. However, accuracy often degrades significantly in weight-only PTQ at sub-4-bit precision, and our analysis identifies two main causes: (1) down-projection matrices are a well-known quantization bottleneck, but maintaining their fidelity often requires extra bit-width; (2) weight quantization induces activation deviations, but effective correction strategies remain underexplored. To address these issues, we propose D$^2$Quant, a novel weight-only PTQ framework that improves quantization from both the weight and activation perspectives. On the weight side, we design a Dual-Scale Quantizer (DSQ) tailored to down-projection matrices, with an absorbable scaling factor that significantly improves accuracy without increasing the bit budget. On the activation side, we propose Deviation-Aware Correction (DAC), which incorporates a mean-shift correction within LayerNorm to mitigate quantization-induced activation distribution shifts. Extensive experiments across multiple LLM families and evaluation metrics show that D$^2$Quant delivers superior performance for weight-only PTQ at sub-4-bit precision. The code and models will be available at https://github.com/XIANGLONGYAN/D2Quant.

Figures (5)

• Figure 1: Performance comparison of weight-only PTQ methods on Qwen-3-8B with 2-bit quantization. D$^2$Quant consistently outperforms all other methods across all evaluation metrics.
• Figure 2: (a) Equivalent transformation between up and down projections in per-channel quantization: smoothing can be applied to down projection, while up projection only introduces channel-wise scaling. (b) Activation deviation at the subsequent LayerNorm after quantizing attention and MLP: quantizing attention causes a notable mean shift, whereas MLP quantization introduces no significant shift.
• Figure 3: Overview of D$^2$Quant. The left panel shows the D$^2$Quant framework, which improves weight-only PTQ at both weight and activation levels. The middle shows the Dual-Scale Quantizer, which introduces an additional scale to refine the down-projection. The right depicts the Deviation-Aware Correction, which mitigates mean shift at post-attention LayerNorm via bias alignment.
• Figure 4: Mean SNR across transformer layers on LLaMA-3-8B. Post-attention LayerNorm exhibits consistently higher signal-to-noise ratios than Pre-LayerNorm, indicating stronger and more structured activation shifts caused by attention quantization.
• Figure 5: Model size and quantization time on LLaMA-3-8B.