FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

TL;DR

FBI-LLM demonstrates that fully binarized LLMs can be trained from scratch to compete with full-precision models across multiple scales using autoregressive distillation. The approach binarizes nearly all linear layers while preserving embeddings and layernorm, and introduces column-wise scaling in FBI-Linear to recover capacity. A full-precision LLaMA2-7B acts as the teacher, guiding the binarized student at each token via a distillation loss with STE-based backpropagation. Experiments on 130M, 1.3B, and 7B models trained on the Amber dataset show favorable perplexity and downstream performance with substantial storage reductions, suggesting practical hardware and deployment benefits. Limitations include a remaining performance gap to FP models, training cost, and current hardware constraints for 1-bit LLMs.

Abstract

This work presents a Fully BInarized Large Language Model (FBI-LLM), demonstrating for the first time how to train a large-scale binary language model from scratch (not the partial binary or ternary LLM like BitNet b1.58) to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs. It achieves this by employing an autoregressive distillation (AD) loss with maintaining equivalent model dimensions (130M, 1.3B, 7B) and training data volume as regular LLM pretraining, while delivering competitive results in terms of perplexity and task-specific effectiveness. Intriguingly, by analyzing the training trajectory, we find that the pretrained weight is not necessary for training binarized LLMs from scratch. This research encourages a new computational framework and may facilitate the future design of specialized hardware tailored for fully 1-bit LLMs. We make all models, code, and training dataset fully accessible and transparent to support further research (Code: https://github.com/LiqunMa/FBI-LLM. Model: https://huggingface.co/LiqunMa/).

Figures (7)

• Figure 1: Perplexity on Wikitext2 of existing binarized LLMs and our FBI-LLMs. FBI-LLMs get similar or lower magnitude of perplexity on similar size of models compared with other binarized LLMs.
• Figure 2: Illustration of the FBI-LLM framework. We take the structure of LLaMA as an example. Left: the LLM block with the proposed FBI-Linear using learnable $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$. Right: our autoregressive distillation and model pertaining procedure.
• Figure 3: Changes in average perplexity and downstream task accuracy during the training of FBI-LLM 7B. The horizontal axis represents the number of Amber data blocks used for training.
• Figure 4: The model performance for different training loss.
• Figure 5: The flip-flop ratio and loss for different training procedures.