BitNet a4.8: 4-bit Activations for 1-bit LLMs

TL;DR

BitNet a4.8 presents a hybrid quantization and sparsification framework that enables $4$-bit activations for $1$-bit LLMs, pairing them with $8$-bit sparsified intermediate states to curb quantization errors from outlier channels. The approach uses BitLinear for $1.58$-bit weights, a two-stage training scheme with STE gradient approximation, and supports $3$-bit KV cache to boost deployment efficiency; FP4/INT4 kernels further accelerate inference. Empirical results on multiple model sizes show BitNet a4.8 achieving performance close to BitNet b1.58 at the same training cost, with substantial sparsity gains (e.g., $44.5\%$ overall sparsity at 7B) and robust low-bit attention performance, including 3-bit KV. The work demonstrates practical gains in latency, memory, and energy for large-scale LLMs, enabling faster, more scalable deployment while maintaining accuracy across zero-shot tasks and perplexity metrics.

Abstract

Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.

Figures (6)

• Figure 1: The overview of BitNet a4.8 with both weight and activation quantization. All the parameters are ternery (i.e., 1.58-bit as in BitNet b1.58 bitnet2). We use a hybrid quantization and sparsification strategy to deal with outlier activations in certain Transformer sub-layers.
• Figure 2: The distribution of the inputs to each projection. The visualization is conducted with a 7B BitNet b1.58 model on a subset of the valid set of C4. For the layers that exhibit Gaussian-like distributions, we employ 4-bit activation quantization. For the layers which distributions are sharp, we adopt Q-Sparse qsparse to perform sparsification on the activations.
• Figure 3: The distribution of the inputs to the output projection of attention with different quantization and sparsification. The visualization is conducted with a 7B BitNet b1.58 model on a subset of the valid set of C4.
• Figure 4: Ablation study on the hybrid quantization and sparsification.
• Figure 5: Ablation study on different quantization or activation function for the inputs to down projection of FFN.