Binary Neural Networks for Large Language Model: A Survey

TL;DR

The survey tackles the resource challenges of large language models by focusing on binary neural networks and 1-bit quantization as a path to substantial memory and compute savings. It centers on BitNet as the pioneering from-scratch 1-bit Transformer approach and surveys subsequent optimizations in weight/activation quantization, KV-cache handling, and novel loss functions, as well as training strategies like autoregressive distillation and mixed-precision schemes. The work analyzes performance, convergence, and energy benefits relative to traditional PTQ/QAT baselines, highlighting the potential and caveats of binarized LLMs for practical deployment. Overall, the paper argues that binary quantization unlocks efficient, scalable LLM deployment across CPUs, GPUs, and edge devices, while outlining key directions for improving training efficiency and model accuracy.

Abstract

Large language models (LLMs) have wide applications in the field of natural language processing(NLP), such as GPT-4 and Llama. However, with the exponential growth of model parameter sizes, LLMs bring significant resource overheads. Low-bit quantization, as a key technique, reduces memory usage and computational demands by decreasing the bit-width of model parameters, activations, and gradients. Previous quantization methods for LLMs have largely employed Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ does not require any retraining of the original model, while QAT involves optimizing precision during training to achieve the best quantization parameters. The BitNet team proposed a radically different approach, where quantization is performed from the start of model training, utilizing low-precision binary weights during the training process. This approach has led to the emergence of many binary quantization techniques for large language models. This paper provides a comprehensive review of these binary quantization techniques. Specifically, we will introduce binary quantization techniques in deep neural networks and further explore their application to LLMs, reviewing their various contributions, implementations, and applications.

Figures (7)

• Figure 1: The forward and backward propagation diagrams of different algorithms, where the upper/lower part represents forward/backward propagation.
• Figure 2: Schematic of BitLinear, where in this diagram, it is assumed that the activations are divided into n groups and the weights are divided into m groups.
• Figure 3: A comparison of weight quantization for different networks, where the blue/green plots represent the results before/after quantization.
• Figure 4: A specific example of weight quantization methods across different networks. The scaling factors for BitNet, BitNet b1.58, and BitNet a4.8 are set to 0.75, while the scaling factors for FBI-LLM and Bi-Mamba are set as follows: $\alpha = [0.7, 0.3, 0.1, -0.5]$ and $\beta = [0.01, 0.04, 0.02, 0.06]$.
• Figure 5: A comparison of activation quantization techniques across various networks. BitNet a4.8 utilizes a hybrid quantization and sparsification strategy to address outlier activations in certain Transformer sub-layers.