Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BitNet b1.58 2B4T Technical Report

Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, Furu Wei

TL;DR

This work tackles the efficiency barrier of open-source LLMs by introducing BitNet b1.58 2B4T, a native 1-bit LLM at 2B parameters trained on 4T tokens. It combines BitNet-specific architectural choices (BitLinear with 1.58-bit weights, absmean quantization to ${-1,0,+1}$, 8-bit activations, RoPE, bias removal, and squared ReLU) with a three-stage training regime (pre-training, SFT, DPO). Evaluation demonstrates competitive performance against open-weight full-precision baselines across language understanding, reasoning, coding, and dialogue, while delivering substantial gains in memory, energy, and latency; comparisons with PTQ and other 1-bit systems highlight BitNet’s efficiency edge. The authors publicly release weights and GPU/CPU inference implementations via open-source channels, enabling broader adoption and further research in resource-constrained settings.

Abstract

We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.

BitNet b1.58 2B4T Technical Report

TL;DR

This work tackles the efficiency barrier of open-source LLMs by introducing BitNet b1.58 2B4T, a native 1-bit LLM at 2B parameters trained on 4T tokens. It combines BitNet-specific architectural choices (BitLinear with 1.58-bit weights, absmean quantization to , 8-bit activations, RoPE, bias removal, and squared ReLU) with a three-stage training regime (pre-training, SFT, DPO). Evaluation demonstrates competitive performance against open-weight full-precision baselines across language understanding, reasoning, coding, and dialogue, while delivering substantial gains in memory, energy, and latency; comparisons with PTQ and other 1-bit systems highlight BitNet’s efficiency edge. The authors publicly release weights and GPU/CPU inference implementations via open-source channels, enabling broader adoption and further research in resource-constrained settings.

Abstract

We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.

Paper Structure

This paper contains 26 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Training
  4. Pre-training
  5. Pre-training
  6. Learning Rate Schedule
  7. Weight Decay Schedule
  8. Pre-training Data
  9. Supervised Fine-tuning (SFT)
  10. SFT Data
  11. Chat Template
  12. Optimization Details
  13. Direct Preference Optimization (DPO)
  14. Training Data
  15. Training Details
  16. ...and 11 more sections

Figures (1)

  • Figure 1: BitNet b1.58 2B4T advances the Pareto frontier defined by leading open-weight LLMs under 3B parameters in terms of performance versus memory, demonstrating superior efficiency.