Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs

Hongyu Wang, Shuming Ma, Furu Wei

TL;DR

Activation outliers hinder 4-bit activation quantization in 1-bit LLMs. The authors propose H-BitLinear, which applies an online Hadamard transform before activation quantization to reshape intermediate activations toward Gaussian-like distributions. BitNet v2 enables native 4-bit activations by replacing attention output projections and FFN down projections with H-BitLinear and training from 8-bit activations before fine-tuning to 4-bit. Empirically, BitNet v2 matches or surpasses prior 1.58-bit models, achieves perplexity similar to 4-bit-a4.8, and offers substantial memory and compute savings for batched inference.

Abstract

Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.

BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs

TL;DR

Activation outliers hinder 4-bit activation quantization in 1-bit LLMs. The authors propose H-BitLinear, which applies an online Hadamard transform before activation quantization to reshape intermediate activations toward Gaussian-like distributions. BitNet v2 enables native 4-bit activations by replacing attention output projections and FFN down projections with H-BitLinear and training from 8-bit activations before fine-tuning to 4-bit. Empirically, BitNet v2 matches or surpasses prior 1.58-bit models, achieves perplexity similar to 4-bit-a4.8, and offers substantial memory and compute savings for batched inference.

Abstract

Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.

Paper Structure

This paper contains 11 sections, 5 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. BitNet v2: Native 4-bit Activations
  3. $\mathcal{H}$-BitLinear
  4. Training
  5. Experiments
  6. Main Results
  7. Comparison with Post-Training Quantization.
  8. Ablation Study
  9. Conclusion
  10. More results
  11. Hyper-parameters

Figures (3)

  • Figure 1: Top: Overview of BitNet v2 and $\mathcal{H}$-BitLinear. Bottom: The distribution of the activation of output projection $\mathbf{W}_{\text{o}}$ in attention and down projection $\mathbf{W}_{\text{down}}$ in FFN. BitNet v2 utilizes $\mathcal{H}$-BitLinear to eliminate the large amount of outlier channels in the intermediate states. The Hadamard transformation reshapes the original sharp distribution into a more Gaussian-like form.
  • Figure 2: The activation distribution of BitNet b1.58 and BitNet v2 with 8-bit activations.
  • Figure 3: The activation distribution of $\mathbf{W}_{\text{down}}$ in FFN and $\mathbf{W}_{\text{o}}$ in attention of BitNet b1.58 and BitNet v2 with 8-bit activations.