Addition is almost all you need: Compressing neural networks with double binary factorization

TL;DR

The paper introduces Double Binary Factorization (DBF), a method to compress dense weight matrices by representing them as the product of two binary sign matrices with scaling vectors, enabling binary-level computation while allowing fine-grained control over compression via the middle dimension. It provides a practical construction, an ADMM-based optimization, and an importance-weighting scheme to preserve critical weights, plus a nonuniform, iterative budget allocation across layers. Through extensive experiments on LLama models, DBF achieves competitive accuracy at low bitrates (1–2 bits) and significant inference speedups (2–3.5x) with potential energy savings, often outperforming or matching state-of-the-art quantization methods at similar compression levels. The work highlights DBF’s flexibility and practical viability on current hardware, along with directions for on-the-fly factorization during fine-tuning and deeper integration with channel-pruning strategies.

Abstract

Binary quantization approaches, which replace weight matrices with binary matrices and substitute costly multiplications with cheaper additions, offer a computationally efficient approach to address the increasing computational and storage requirements of Large Language Models (LLMs). However, the severe quantization constraint ($\pm1$) can lead to significant accuracy degradation. In this paper, we propose Double Binary Factorization (DBF), a novel method that factorizes dense weight matrices into products of two binary (sign) matrices, each accompanied by scaling vectors. DBF preserves the efficiency advantages of binary representations while achieving compression rates that are competitive with or superior to state-of-the-art methods. Specifically, in a 1-bit per weight range, DBF is better than existing binarization approaches. In a 2-bit per weight range, DBF is competitive with the best quantization methods like QuIP\# and QTIP. Unlike most existing compression techniques, which offer limited compression level choices, DBF allows fine-grained control over compression ratios by adjusting the factorization's intermediate dimension. Based on this advantage, we further introduce an algorithm for estimating non-uniform layer-wise compression ratios for DBF, based on previously developed channel pruning criteria. Code available at: https://github.com/usamec/double_binary

Figures (3)

• Figure 1: Left: Schematic drawing of computation in Double binary factorization. $X$ is the layer input in FP16 format, $W$ is the original weight matrix. $A$ and $B$ consist of $\pm 1$ elements, $a,m,b$ are vectors in FP16 format. We omit matrix transpositions for simplicity. Right: Comparison of our Double binary factorization (DBF) with previously proposed network compression methods on Llama2-7B.
• Figure 2: Comparison of weight importance (calculated as a product of input and output importance) vs. approximation error on 7.self_attn.k_proj of Llama3-8B. Blue points are actual matrix elements, and the orange line is a smoothed version.
• Figure 3: Comparison of number of bits vs. approximation error on two layers of Llama3-8B.