BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

TL;DR

BiLLM pushes the frontier of post-training quantization by achieving near 1-bit weight representations for pretrained LLMs. It jointly employs Hessian-based salient-weight selection with a binary residual approach and distribution-aware splitting (concentrated vs. sparse) for non-salient weights, enabling high-accuracy binary inference across multiple LLM families. The method demonstrates state-of-the-art perplexities at ultra-low bit-widths (e.g., ~8.4 on WikiText2 with LLaMA2-70B) and practical quantization times on a single GPU, signaling strong potential for edge deployment. These contributions offer a practical, scalable path to dramatically reduce memory and compute for large language models without retraining.

Abstract

Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at https://github.com/Aaronhuang-778/BiLLM.

Figures (15)

• Figure 1: Perplexity of LLaMA-13B on WikiText2 under different bit-widths. Round-to-nearest (RTN), GPTQ, and PB-LLM (10% weight of INT8) suffer accuracy loss at ultra-low bits, facing the sharply increasing perplexity ($\downarrow$). BiLLM demonstrates exceptional performance under binarization.
• Figure 2: The Hessian metrics (sensitivity) and magnitude (value) of weights in LLMs. The weights of different layers in LLMs are characterized by bell-shaped distribution, accompanied by a few salient values.
• Figure 3: Schematic of the PTQ binarization framework for LLMs. The left side shows the structure of the Transformer block after binarization. The right side shows the binarization process of BiLLM, which consists of two parts, Residual Approximation for salient weights and Bell-shaped Splitting for non-salient weights.
• Figure 4: Illustration of salient weight binarization. The $\mathbf{B}_1$ binarized from salient weight is made into a residual with the original value and then binarized again to obtain $\mathbf{B}_2$.
• Figure 5: Distribution and splitting schematic of the $4^{th}$ projection layer in LLaMA2-7B. The top 5% of the Hessian elements are orange, and the optimal break-point divides the non-salient weights into sparse and concentrated areas.