Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiBERT: Accurate Fully Binarized BERT

Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, Xianglong Liu

TL;DR

BiBERT tackles the challenge of fully binarized BERT by identifying two core bottlenecks: information degradation in binarized attention and optimization-direction mismatch during distillation. It introduces Bi-Attention, an entropy-maximizing binarized attention mechanism implemented via a Bool-based binarizer and BAMM for efficient bitwise computation, and Direction-Matching Distillation (DMD), which aligns distillation targets using Q/K/V-based similarity matrices to correct optimization directions. Empirical results on GLUE show BiBERT surpassing prior ultra-low-bit quantized BERTs, with substantial gains in both accuracy and efficiency (e.g., 56.3x FLOPs and 31.2x model size reductions). The work demonstrates a viable route for deploying fully binarized transformers on resource-constrained devices without sacrificing baseline performance, marking a significant step in extreme model compression for NLP.

Abstract

The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.

BiBERT: Accurate Fully Binarized BERT

TL;DR

BiBERT tackles the challenge of fully binarized BERT by identifying two core bottlenecks: information degradation in binarized attention and optimization-direction mismatch during distillation. It introduces Bi-Attention, an entropy-maximizing binarized attention mechanism implemented via a Bool-based binarizer and BAMM for efficient bitwise computation, and Direction-Matching Distillation (DMD), which aligns distillation targets using Q/K/V-based similarity matrices to correct optimization directions. Empirical results on GLUE show BiBERT surpassing prior ultra-low-bit quantized BERTs, with substantial gains in both accuracy and efficiency (e.g., 56.3x FLOPs and 31.2x model size reductions). The work demonstrates a viable route for deploying fully binarized transformers on resource-constrained devices without sacrificing baseline performance, marking a significant step in extreme model compression for NLP.

Abstract

The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.
Paper Structure (34 sections, 8 theorems, 51 equations, 8 figures, 7 tables)

This paper contains 34 sections, 8 theorems, 51 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Building a Fully Binarized BERT Baseline
  3. Binarized BERT Architecture
  4. Distillation for Binarized BERT
  5. The Rise of BiBERT
  6. Bottlenecks of Fully Binarized BERT Baseline
  7. Bi-Attention
  8. Information Degradation in Attention Structure
  9. Bi-Attention for Maximum Information Entropy
  10. Direction-Matching Distillation
  11. Direction Mismatch
  12. Direction-Matching Distillation for Accurate Optimization
  13. Experiments
  14. Ablation Study
  15. Comparison with SOTA Methods
  16. ...and 19 more sections

Key Result

Theorem 1

Given $\mathbf A\in \mathbb{R}^{k}$ with Gaussian distribution and the variable $\mathbf{\hat{B}_{A}}^s$ generated by $\mathbf{\hat{B}_{\mathbf{w}}}^A =\operatorname{sign}(\operatorname{softmax}(\mathbf{A})-\tau)$, the threshold $\tau$, which maximizes the information entropy $\mathcal{H}(\mathbf{\h

Figures (8)

  • Figure 1: Accuracy vs. FLOPs & size. Our BiBERT enjoys most computation and storage savings while surpassing SOTA quantized BERTs on GLUE benchmark with low bit activation.
  • Figure 2: Overview of our BiBERT, applying Bi-Attention structure for maximizing representation information and Direction-Matching Distillation (DMD) scheme for accurate optimization.
  • Figure 3: Analysis of bottlenecks from architecture and optimization perspectives. We report the accuracy of binarized BERT on SST-2 and QQP tasks about (a) replace full-precision structure, (b) exclude one distillation knowledge.
  • Figure 4: Attention-head view for (a) full-precision BERT, (b) fully binarized BERT baseline, and (c) BiBERT for same input. BiBERT with Bi-Attention shows similar behavior with the full-precision model, while baseline suffers indistinguishable attention for information degradation. The visualization tools is adapted from vig2019multiscale.
  • Figure 5: Visualization of the direction mismatch of one head throughout training. The full binarized BERT baseline distills the attention score which has severe direction mismatch, while the knowledge (take queries as example in (b)) used in our BiBERT significantly alleviates mismatch.
  • ...and 3 more figures

Theorems & Definitions (13)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Proposition 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 4
  • ...and 3 more