Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization

Jacob Nielsen, Lukas Galke, Peter Schneider-Kamp

TL;DR

This work explores 1.58-bit training in transformer-based language models, namely encoder-only and encoder-decoder models, and shows that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.

Abstract

Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.

When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization

TL;DR

This work explores 1.58-bit training in transformer-based language models, namely encoder-only and encoder-decoder models, and shows that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.

Abstract

Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.

Paper Structure

This paper contains 27 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background: b1.58 Quantization
  3. BitNet in Non-Transformer Models
  4. Can 1.58-bit models solve X-OR?
  5. Setup
  6. Results
  7. BitNet in Multilayer Perceptrons
  8. Setup
  9. Results
  10. BitNet in Graph Neural Networks
  11. Setup
  12. Results
  13. Summary and Interim Discussion
  14. BitNet in Transformer Architectures
  15. Encoder Models
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Scaling Behavior of 16-bit and 1.58-bit (median) BERT Encoder Train Loss over $150$K train-steps. Smoothing applies a Savitzky-Golay filter with a window size of 1000 and a polynomial order of 2.
  • Figure 2: Scaling Behavior of 16-bit and b1.58 (median) employed in T5v1.1-Base Encoder-Decoder Architecture. 16-bit and b1.58 employed throughout the network over increasing hidden size. d_ff denotes the hidden size of the FFN within each encoder and decoder stack.
  • Figure 3: Regularization Effect of b1.58 (median), OLMo 1B model trained on the internal dataset described in Section \ref{['sec:transformers:encoders']}. Smoothing is applied using a Savitzky-Golay filter with a window size of 1000 and polynomial order 2.