BitNet Distillation

TL;DR

The paper tackles the challenge of deploying large LLMs on resource-constrained devices by aiming for $1.58$-bit quantization. It proposes BitDistill, a three-stage distillation framework—modeling refinement with SubLN, continued pre-training on a compact corpus, and logits plus attention distillation—to bridge the gap between ultra-low-bit weights and full-precision performance. Across multiple tasks and model sizes, BitDistill achieves performance comparable to FP16 baselines while delivering up to $10×$ memory savings and $2.65×$ faster CPU inference, and demonstrates robustness to different backbones and quantization methods. These results provide a scalable, deployment-friendly approach for efficient LLMs on CPU-bound and edge hardware.

Abstract

In this paper, we present BitNet Distillation (BitDistill), a lightweight pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream tasks, achieving strong task-specific performance with minimal computational cost. Specifically, BitDistill incorporates three key techniques: the SubLN module, as introduced in BitNet; multi-head attention distillation, based on MiniLM; and continual pre-training, which serves as a crucial warm-up step to mitigate the scalability issue of the performance gap between finetuned full-precision and 1.58-bit LLMs on specific tasks. Experimental results show that BitDistill achieves performance comparable to the full-precision counterpart models across model size, while enabling up to 10x memory savings and 2.65x faster inference on CPUs. Code is available at https://github.com/microsoft/BitNet.

Figures (3)

• Figure 1: Performance on downstream tasks across model size, with inference speed and memory efficiency comparison. We observed that directly finetuning full-precision LLMs into 1.58-bit LLMs (denoted as 1.58-bit BitNet-SFT) leads to a notable performance gap compared to the FP16 baseline, and this gap remains or even widens as the model size increases. In contrast, BitDistill preserves scalability, resulting in performance comparable to full-precision counterparts across all model size, while reducing $10\times$ memory usage and $2.65\times$ faster inference on CPUs.
• Figure 2: Visualization of model weights. The top two rows show the quantized weights of BitNet trained from scratch along with their corresponding FP16 distributions. The bottom two rows show the quantized weights of BitNet after loading weights from LLMs and performing continued training (stage-2 in § \ref{['sec: training']}), together with their corresponding FP16 distributions.
• Figure 3: Analysis of SubLN, layer selection for Eq. \ref{['eq: attn distill']} and teacher selection over training steps. (a) Fine-tuning existing LLMs into 1.58-bit BitNet with SubLN yields better performance and faster convergence. (b) Comparison of MNLI accuracies obtained by distilling different layers on Qwen3-0.6B. (c) Comparison of MNLI accuracies obtained by distilling Qwen3-0.6B with different size of FP16 teachers.