SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights

TL;DR

SINQ tackles the degradation of post-training quantization at very low bit-widths by introducing a dual-scaling parameterization that applies scales along both matrix dimensions and a Sinkhorn-Knopp inspired normalization to minimize a matrix imbalance objective. The method operates independently per linear layer and can be paired with calibration or non-uniform quantization, yielding strong improvements on Qwen3 and DeepSeek-V2.5 while maintaining practical quantization speed. Across extensive experiments, SINQ outperforms calibration-free baselines in perplexity and flip rates, and remains compatible with activation-aware calibration and NF4. Its architecture-agnostic design and fast runtime make it a practical pathway toward calibration-free low-precision LLM weights for diverse model families.

Abstract

Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision. Still, current methods show perplexity degradation at bit-widths less than or equal to 4, partly because representing outliers causes precision issues in parameters that share the same scales as these outliers. This problem is especially pronounced for calibration-free, uniform quantization methods. We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn-Knopp-style algorithm that finds scales to normalize per-row and per-column variances, thereby minimizing a novel per-matrix proxy target for quantization: the matrix imbalance. Our method has no interactions between layers and can be trivially applied to new architectures to quantize any linear layers. We evaluate our method on the Qwen3 model family and DeepSeek-V2.5. SINQ improves WikiText2 and C4 perplexity significantly against uncalibrated uniform quantization baselines and can be further enhanced by combining it with calibration and non-uniform quantization levels. Code to reproduce the results of this work and to easily quantize models using SINQ is available at https://github.com/huawei-csl/SINQ.

Figures (6)

• Figure 1: If we have scales along both dimensions of a matrix that is to be quantized, we can trade off the impact of outliers between rows and columns, which is impossible in single-scale quantization. Left: Conceptual illustration of quantization error distributions with single or dual-scaling. Right: Example on small matrix.
• Figure 2: Results on Qwen3-1.7B. Minimizing the imbalance with our algorithm (a and b) decreases both the imbalance and the kurtosis. Minimizing the kurtosis directly with gradient descent (d and e) yields lower kurtosis, but causes a large imbalance; note the log-scale on (d). Finally (c) and (f) show the end-to-end perplexity on WikiText2 and per-layer RTN MSE improvement when optimizing imbalance or kurtosis, respectively.
• Figure 3: Pareto plot in terms of memory vs. WikiText2 perplexity for Qwen3-0.6B to 32B for different uncalibrated quantization methods. (a) compares different 4-bit methods (including FP4, INT4, and NF4 where available). The maximum distance from the 4-bit pareto front of our method is $<0.01$ppl. Note that the difference to the baseline is small. (b) allows bit widths of 4, 6, 8. For 8-bit quantization we inlcude LLM.int8() from llmint8 as a reference method. Both plots include the BF16 model as a baseline. For these plots we allow group sizes 64 and 128 for all methods.
• Figure 4: Ablation experiments in the form of memory-perplexity Pareto-fronts across the Qwen3 family on WikiText2. (a) Auxiliary variable precision (b) Tiling dimension (c) With/without shifts.
• Figure 5: Distribution of quantization times for each method for Qwen3-32B.