Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization

Yuma Ichikawa, Yoshihiko Fujisawa, Yudai Fujimoto, Akira Sakai, Katsuki Fujisawa

TL;DR

This work targets extreme low-bit quantization of large language models by addressing a fundamental bottleneck in Double Binary Factorization (DBF): after demodulation, all magnitudes collapse to a single envelope, limiting expressivity. The authors propose Multi-Envelope DBF (MDBF), which preserves a shared 1-bit sign base but relaxes the envelope to a rank-$l$ form, enabling multiple magnitude modes without changing the deployment-friendly binary inference path. They develop a practical layer-wise PTQ pipeline with a closed-form MSVID-based initialization and an ADMM-inspired refinement that enforces the rank-$l$ envelope, and demonstrate consistent improvements in perplexity and zero-shot accuracy across LLaMA and Qwen at matched bits-per-weight. MDBF allocates capacity to magnitude modeling rather than sign diversity, enabling more faithful reconstruction under extreme quantization and improving deployment efficiency for large-scale models. The results suggest that adopting a rank-$l$ envelope is a more effective utilization of the limited bit budget than simply increasing the number of sign patterns.

Abstract

For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. However, the scaling parameters of DBF are too restrictive; after factoring out signs, all rank components share the same magnitude profile, resulting in performance saturation. We propose Multi-envelope DBF (MDBF), which retains a shared pair of 1-bit sign bases but replaces the single envelope with a rank-$l$ envelope. By sharing sign matrices among envelope components, MDBF effectively maintains a binary carrier and utilizes the limited memory budget for magnitude expressiveness. We also introduce a closed-form initialization and an alternating refinement method to optimize MDBF. Across the LLaMA and Qwen families, MDBF enhances perplexity and zero-shot accuracy over previous binary formats at matched bits per weight while preserving the same deployment-friendly inference primitive.

More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization

TL;DR

This work targets extreme low-bit quantization of large language models by addressing a fundamental bottleneck in Double Binary Factorization (DBF): after demodulation, all magnitudes collapse to a single envelope, limiting expressivity. The authors propose Multi-Envelope DBF (MDBF), which preserves a shared 1-bit sign base but relaxes the envelope to a rank- form, enabling multiple magnitude modes without changing the deployment-friendly binary inference path. They develop a practical layer-wise PTQ pipeline with a closed-form MSVID-based initialization and an ADMM-inspired refinement that enforces the rank- envelope, and demonstrate consistent improvements in perplexity and zero-shot accuracy across LLaMA and Qwen at matched bits-per-weight. MDBF allocates capacity to magnitude modeling rather than sign diversity, enabling more faithful reconstruction under extreme quantization and improving deployment efficiency for large-scale models. The results suggest that adopting a rank- envelope is a more effective utilization of the limited bit budget than simply increasing the number of sign patterns.

Abstract

For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. However, the scaling parameters of DBF are too restrictive; after factoring out signs, all rank components share the same magnitude profile, resulting in performance saturation. We propose Multi-envelope DBF (MDBF), which retains a shared pair of 1-bit sign bases but replaces the single envelope with a rank- envelope. By sharing sign matrices among envelope components, MDBF effectively maintains a binary carrier and utilizes the limited memory budget for magnitude expressiveness. We also introduce a closed-form initialization and an alternating refinement method to optimize MDBF. Across the LLaMA and Qwen families, MDBF enhances perplexity and zero-shot accuracy over previous binary formats at matched bits per weight while preserving the same deployment-friendly inference primitive.
Paper Structure (44 sections, 6 theorems, 43 equations, 2 figures, 4 tables)

This paper contains 44 sections, 6 theorems, 43 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Notation
  4. Preliminaries
  5. Low-Rank Approximation
  6. Double Binary Factorization
  7. Single-Envelope Constraint.
  8. Inference.
  9. Residual Compensation.
  10. Method
  11. Bottleneck: Single-Envelope Constraint
  12. Multi-Envelope Double Binary Factorization
  13. Multi-Envelope Structure
  14. Demodulation under Fixed Sign Mask.
  15. Rank-$l$ Envelope Class.
  16. ...and 29 more sections

Key Result

theorem 1

Let $U\in\mab{R}^{N \times R}$ fix a sign mask $S\in\{\pm 1\}^{N \times R}$. Let $\sigma_1(E_S(U))\ge \cdots \ge \sigma_{\min(N,R)}(E_S(U))\ge 0$ be the singular values of $E_S(U)=S\odot U$. Fix an integer $l$ such that $1\le l\le \min(N, R)$. Then Moreover, one minimizer is where $\mathrm{TSVD}_l(\cdot)$ denotes the rank-$l$ truncated SVD, which provides the best approximation in the Frobenius

Figures (2)

  • Figure 1: Layer-wise reconstruction error vs. envelope rank and decomposition depth (LLaMA2 7B). We conduct experiments under a 1.5 bit quantization setting and report the relative Frobenius error $\|W-\widehat{W}\|_{F}/\|W\|_{F}$ of MDBF as a function of the envelope rank $l$ and the decomposition depth $P$, evaluated on three representative Transformer blocks of layers $0$, $15$, and $30$ and central attention/MLP projections. Across layers and modules, larger $l$, which increases magnitude expressivity while sharing sign bases, consistently lowers reconstruction error. In contrast, larger $P$, which adds extra decompositions and sign bases, often worsens reconstruction at matched bits per weight. The best configuration in this sweep is $(l^\ast, P^\ast)=(16,1)$.
  • Figure 2: Entropy-based effective rank across Transformer layers for LLaMA2 13B. We report the average effective rank of the full weight matrices $W$, their binarized signs $\mathrm{sign}(W)$, and the demodulated envelopes $|U|$ and $|V|$. The envelopes remain consistently above rank one, indicating multiple magnitude modes and motivating the relaxation of the single-envelope constraint.

Theorems & Definitions (14)

  • definition 1
  • theorem 1
  • definition 2
  • lemma 1
  • proof
  • lemma 2
  • proof
  • theorem 2
  • proof
  • remark 1
  • ...and 4 more