STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs

TL;DR

This paper introduces a novel Standardized Importance (SI) metric, which considers weight magnitude and input feature norm to more accurately assess weight significance and proposes a layer-wise approach, allowing different layers of the LLM to be sparsified with varying N:M ratios, thereby balancing compression and accuracy.

Abstract

In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memory-bound nature during the inference stage hinders the adoption of resource-constrained devices. Reducing weights to 1-bit precision through binarization substantially enhances computational efficiency. We observe that some weights in binarized LLMs can be randomly flipped without significant performance degradation, suggesting the potential for further compression. To exploit this, our STBLLM employs an N:M sparsity technique to achieve structural binarization of the weights. Specifically, we introduce a novel Standardized Importance (SI) metric, which considers weight magnitude and input feature norm to more accurately assess weight significance. Then, we propose a layer-wise approach, allowing different layers of the LLM to be sparsified with varying N:M ratios, thereby balancing compression and accuracy. Furthermore, we implement a fine-grained grouping strategy for less important weights, applying distinct quantization schemes to sparse, intermediate, and dense regions. Finally, we design a specialized CUDA kernel to support structural binarization. We conduct extensive experiments on LLaMA-1/2/3, OPT family, and Mistral to evaluate the effectiveness of STBLLM. The results demonstrate that our approach performs better than other compressed binarization LLM methods while significantly reducing memory requirements.

Figures (12)

• Figure 1: The impact of random flipping non-salient binarized weights on accuracy in a 1-Bit LLaMA-2-7B. The x-axis represents the percentage of binarized weights flipped from -1 to 1 or vice versa. As the ratio increases, the accuracy does not decline significantly, indicating redundancy in the 1-bit representation.
• Figure 2: The perplexity of LLaMA-1-13B on the Wikitext2 under different bit-widths. RTN and GPTQ Frantar2022GPTQAP show a drastic performance drop at ultra-low bit-widths. Our proposed STBLLM achieves higher performance compared to BiLLM at sub 1-bit widths.
• Figure 3: (a) PTQ framework in Structured Binarized LLM (STBLLM). We apply structured binarization to all of the weights. (b) Structured Binarized Weight Computation Procedure. We first perform N:M structure pruning to pre-trained weight (here N=2, M=4), then perform binarization by assign weight to salient and non-salient one. (c) Trisection partition for Symmetric Gaussian Distribution of Non-salient Weight. (d) Illustration of Weight Standardization on LLaMA-2-7B.
• Figure 4: (a) Runtime and throughput comparison across sequence lengths for ours and ABQ-LLM. (b) Perplexity comparison across model sizes under 2:4 setting for LLaMA-1/2.
• Figure 5: Structured Pruned Matrix 4-Group Index Representation for 2:4 Structured Sparsity Acceleration.