Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARB-LLM: Alternating Refined Binarizations for Large Language Models

Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, zhongchao shi, Linghe Kong, Yulun Zhang, Xiaokang Yang

TL;DR

ARB-LLM presents a distribution-aware 1-bit PTQ framework for large language models by iteratively refining binarization parameters to reduce the gap between binarized and full-precision weights. It extends the basic ARB method with calibration data (ARB-X) and row–column scaling (ARB-RC), and introduces a Column-Group Bitmap (CGB) to efficiently partition weights for higher fidelity. Empirical results across OPT, LLaMA, and Vicuna families show ARB-LLM$_\text{RC}$ and ARB-LLM$_\text{X}$ outperform state-of-the-art binary PTQ methods and, in some cases, surpass FP16 accuracy at the same model size, with reasonable time and memory overhead. The work provides concrete formulas for parameter updates, theoretical insights on error reduction, and practical guidance for deploying highly compressed LLMs on limited hardware.

Abstract

Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_\text{X}$ and ARB-LLM$_\text{RC}$ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$_\text{RC}$ is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM.

ARB-LLM: Alternating Refined Binarizations for Large Language Models

TL;DR

ARB-LLM presents a distribution-aware 1-bit PTQ framework for large language models by iteratively refining binarization parameters to reduce the gap between binarized and full-precision weights. It extends the basic ARB method with calibration data (ARB-X) and row–column scaling (ARB-RC), and introduces a Column-Group Bitmap (CGB) to efficiently partition weights for higher fidelity. Empirical results across OPT, LLaMA, and Vicuna families show ARB-LLM and ARB-LLM outperform state-of-the-art binary PTQ methods and, in some cases, surpass FP16 accuracy at the same model size, with reasonable time and memory overhead. The work provides concrete formulas for parameter updates, theoretical insights on error reduction, and practical guidance for deploying highly compressed LLMs on limited hardware.

Abstract

Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM and ARB-LLM respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM.
Paper Structure (15 sections, 2 theorems, 14 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 15 sections, 2 theorems, 14 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Network Binarization
  4. Large Language Model Quantization
  5. Method
  6. Alternating Refined Binarization (ARB)
  7. ARB with Calibration Data (ARB-X)
  8. ARB along Row-Column Axes (ARB-RC)
  9. Column-Group Bitmap (CGB)
  10. Experiments
  11. Setup
  12. Main Results
  13. Ablation Study
  14. Time and Memory Analyses
  15. Conclusion

Key Result

Theorem 1

For any $\tau\geq 0$, Algorithm alg1 achieves a quantization error $\mathcal{L}_1^{\tau}$ satisfying where $\alpha^0$ and $\mu^0$ denote the initial scaling factor and mean respectively, $\alpha^{\tau}$, $\mu^{\tau}$, and $\mathcal{L}_1^{\tau}$ represent the scaling factor, mean, and quantization error after the $\tau$-th iteration respectively.

Figures (6)

  • Figure 1: OPT performance on 7 zero-shot Question Answering (QA) datasets. Our ARB-LLM$_\text{RC}$ outperforms the same-size FP16 models.
  • Figure 2: Distribution shift between the mean of binarized and full-precision weights. Top: distribution shift of one row. Bottom: distribution shifts of multiple rows. Each row represents a top view of the corresponding upper image.
  • Figure 3: Left: Full-precision weights exhibit column-wise deviations. Right: BiLLM huang2024billm with row-wise binarization smooths the deviations. Our ARB-RC with row-column-wise binarization effectively preserves them.
  • Figure 4: Overview of our ARB series. ARB: alternating refine mean, row scaling factor, and binarized matrix. ARB-X: introducing calibration data into the update of binarization parameters. ARB-RC: alternating refine row and column scaling factors.
  • Figure 5: Comparison between BiLLM and our combination of column and group bitmaps.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2