PB-LLM: Partially Binarized Large Language Models

TL;DR

This work tackles the memory bottleneck of large language models by exploring extreme weight quantization through Partially-Binarized LLMs (PB-LLM). By identifying and preserving a small set of salient weights in higher precision, PB-LLM enables substantial bit-width reductions while maintaining linguistic reasoning, achieved via PTQ (PB-GPTQ) and QAT (salient-weight freezing with optimal binary-weight scaling). The approach demonstrates that Hessian-guided or magnitude-based saliency, combined with analytical scaling and selective freezing, can recover or preserve performance at low bit-widths, outperforming naïve binarization strategies. The findings offer practical pathways for deploying ultra-compressed LLMs with reduced memory and computational demands, validated on OPT-1.3B and LLaMA-7B with publicly shared code.

Abstract

This paper explores network binarization, a radical form of quantization, compressing model weights to a single bit, specifically for Large Language Models (LLMs) compression. Due to previous binarization methods collapsing LLMs, we propose a novel approach, Partially-Binarized LLM (PB-LLM), which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quantized LLMs. Specifically, our exploration first uncovers the ineffectiveness of naive applications of existing binarization algorithms and highlights the imperative role of salient weights in achieving low-bit quantization. Thus, PB-LLM filters a small ratio of salient weights during binarization, allocating them to higher-bit storage, i.e., partially-binarization. PB-LLM is extended to recover the capacities of quantized LMMs, by analyzing from the perspective of post-training quantization (PTQ) and quantization-aware training (QAT). Under PTQ, combining the concepts from GPTQ, we reconstruct the binarized weight matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PB-LLM in low-bit. Under QAT, we freeze the salient weights during training, explore the derivation of optimal scaling factors crucial for minimizing the quantization error, and propose a scaling mechanism based on this derived scaling strategy for residual binarized weights. Those explorations and the developed methodologies significantly contribute to rejuvenating the performance of low-bit quantized LLMs and present substantial advancements in the field of network binarization for LLMs.The code is available at https://github.com/hahnyuan/BinaryLLM.

Figures (7)

• Figure 1: (a) We introduce Partially-Binarized Large Language Model (PB-LLM), where a small subset of the weights of the LLM are frozen and preserved with higher bit precision, while the remaining weights are binarized utilizing an optimal scaling factor strategy; (b) By using PB-LLM, an extremely low-bit LLM can be acquired efficiently (i.e., quantization-aware training converges quickly) while maintaining its language reasoning capabilities.
• Figure 2: We implement five renowned binarization methods on LLMs and assess the resultant binarized LLMs across seven zero-shot common sense reasoning tasks. Random represents the hypothetical worst baseline, indicating random guesses, while FP stands as the optimal baseline, representing full-precision OPT-1.3B. The exact values corresponding to this radar graph are detailed in the Appendix.
• Figure 3: Distribution of $5\%$ salient weight.
• Figure 4: Variation in overall bit number $N_{bit}$ with the ratio of the salient weights $r_{binary}$, where salient weights are stored in 8-bit.
• Figure 5: Training Loss Curves: When only $2\%$ of weights are retained in their un-binarized state, the training loss converges more swiftly.