Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models

TL;DR

This work addresses the inefficiency of uniform post-training quantization for large language models by introducing Benford-Quant (BenQ), a data-free, log-spaced codebook guided by Benford's Law. The method selectively quantizes transformer transformational layers while preserving stability-critical components in higher precision, motivated by empirical Benford-like statistics in weights and a log-domain rationale from multiplicative training dynamics. Across small to mid-sized models, BenQ improves perplexity at 3–4 bits and remains competitive for larger LLMs, with ablations confirming the advantage of log-spaced grids over linear non-uniform alternatives. The approach is hardware-friendly and drop-in compatible with other quantization techniques, offering a practical path toward aggressive few-bit deployment while inviting hybridization with activation-aware quantization methods for further gains.

Abstract

The rapid growth of Large Language Models (LLMs) intensifies the need for effective compression, with weight quantization being the most widely adopted technique. Standard uniform quantizers assume that parameters are evenly distributed, an assumption at odds with the highly skewed distributions observed in practice. We propose Benford-Quant, a simple, data-free non-uniform quantizer inspired by Benford's Law, which predicts that leading digits follow a logarithmic distribution. Benford-Quant replaces the uniform grid with a log-spaced codebook, dedicating more resolution to the frequent small-magnitude weights. We provide both theoretical intuition and empirical evidence: (i) weights in transformer transformational layers adhere closely to Benford statistics, while normalization layers systematically deviate; (ii) on Small Language Models (SLMs), Benford-Quant consistently improves perplexity, reducing 4-bit perplexity on Gemma-270M by more than 10%; and (iii) on larger LLMs, it remains competitive, with differences explained by over-parameterization effects. Our results indicate that incorporating a Benford-inspired prior into quantization grids is a low-cost modification that yields accuracy gains in aggressive few-bit regimes. Although it is not able to surpass the state of the art in tasks such as perplexity and LAMBADA, the Benford-Quant approach can be hybridized with other quantization methods-such as SmoothQuant and Activation-Aware Quantization-without major pipeline modification, potentially improving their performance.

Figures (14)

• Figure 1: The dichotomy of Benford's Law compliance in Llama-3-8B. (a) The weights of a transformational linear layer strongly adhere to the Benford distribution. (b) In contrast, the weights of a 'LayerNorm' layer systematically violate the law, with their first digits overwhelmingly concentrated on a single value. Conclusion: This analysis provides the empirical motivation for both our log-uniform grid and our selective quantization strategy.
• Figure 2: Comparison of Benford’s Law non-compliance across weight families for two models. (a) BLOOM-3B shows consistently higher deviations, particularly in LayerNorm weights. (b) Gemma-3-270M exhibits overall lower deviations, although LayerNorm remains the dominant source of non-compliance. Lines denote the mean MAD per family, with markers indicating the median.
• Figure 3: Benford compliance across layer types. Comparison of digit distributions between two models (Gemma-3-270M vs. BLOOM-3B). Attention and MLP layers exhibit closer adherence to Benford’s law, while other layers diverge more strongly.
• Figure 4: Perplexity vs Parameters. Results for OPT and BLOOM families across bit-widths. Our method (Benford) compared against reported GPTQ baselines. FP16 is shown as reference.
• Figure 5: A numerical example of BenQ in a 2-bit scenario. Our main contribution resides on the log-uniform quantization levels.