Matmul or No Matmul in the Era of 1-bit LLMs

TL;DR

This paper investigates the practical benefits of 1-bit LLMs, noting that extreme quantization typically targets projection layers while leaving attention heads at higher precision. By adapting Amdahl's Law to LLMs, the authors quantify how partial improvements translate to overall model speedups across cloud and edge hardware, using extensive simulations of 13 LLMs and two TPU configurations with OS dataflow. Key findings show that the impact of 1-bit LLMs is highly model- and hardware-dependent: small models gain modest improvements, medium models benefit from combined algorithmic and hardware strategies, and large models can see major throughput gains as most computation shifts to MatMul-free or quantized operations. The work provides a practical roadmap for prioritizing hardware and algorithmic developments, emphasizing projection-layer optimization for edge devices and a balanced approach for larger cloud deployments. The analysis emphasizes that memory traffic is often dominated by projection layers, suggesting targeted memory hierarchy design and data placement to maximize gains in real-world 1-bit LLM deployments.

Abstract

The advent of 1-bit large language models (LLMs) has attracted considerable attention and opened up new research opportunities. However, 1-bit LLMs only improve a fraction of models by applying extreme quantization to the projection layers while leaving attention heads unchanged. Therefore, to avoid fundamentally wrong choices of goals in future research, it is crucial to understand the actual improvements in computation and memory usage that 1-bit LLMs can deliver. In this work, we present an adaptation of Amdahl's Law tailored for the 1-bit LLM context, which illustrates how partial improvements in 1-bit LLMs impact overall model performance. Through extensive experiments, we uncover key nuances across different model architectures and hardware configurations, offering a roadmap for future research in the era of 1-bit LLMs.

Figures (12)

• Figure 1: The typical architecture of decoder-only LLM models and its underlying operations. The tokenization and embedding layers are not shown in the figure.
• Figure 2: The 1-bit LLMs divide the model into two portions: attention heads with MatMul operations (shown in red) and MatMul-free projection layers (shown in green).
• Figure 3: The overall architecture of the TPU that is designed for accelerating LLMs, featuring dedicated hardware to support nonlinear operations.
• Figure 4: Fraction of Matmul-free operations in the OPT models deployed on the cloud setup.
• Figure 5: Amdhal's Law of LLMs for cloud deployment scenario. The dashed lines and solid lines show the effect of partial improvement in projection layers and partial improvement in attention layers, respectively.