BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

TL;DR

BitMoD tackles the memory bottleneck of LLM deployment by proposing an algorithm-hardware co-design that quantizes weights to very low precision using per-group data-type adaptation. It introduces extended FP3/FP4 data types with special values to add asymmetry and extra resolution, coupled with a per-group adaptation strategy and efficient bit-serial dequantization. The hardware realization uses a unified bit-serial representation and a BitMoD PE to support multiple data types, delivering substantial speedups and energy savings over FP16 baselines and prior accelerators like ANT and OliVe, while maintaining high accuracy for both discriminative and generative tasks. The framework is compatible with existing software quantization methods (AWQ, OmniQuant, SmoothQuant), enabling synergistic improvements for edge-oriented LLM inference.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders their deployment. In this paper, we improve the accessibility of LLMs through BitMoD, an algorithm-hardware co-design solution that enables efficient LLM acceleration at low weight precision. On the algorithm side, BitMoD introduces fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights. Through the careful design of these new data types, BitMoD is able to quantize LLM weights to very low precision (e.g., 4 bits and 3 bits) while maintaining high accuracy. On the hardware side, BitMoD employs a bit-serial processing element to easily support multiple numerical precisions and data types; our hardware design includes two key innovations: First, it employs a unified representation to process different weight data types, thus reducing the hardware cost. Second, it adopts a bit-serial dequantization unit to rescale the per-group partial sum with minimal hardware overhead. Our evaluation on six representative LLMs demonstrates that BitMoD significantly outperforms state-of-the-art LLM quantization and acceleration methods. For discriminative tasks, BitMoD can quantize LLM weights to 4-bit with $<\!0.5\%$ accuracy loss on average. For generative tasks, BitMoD is able to quantize LLM weights to 3-bit while achieving better perplexity than prior LLM quantization scheme. Combining the superior model performance with an efficient accelerator design, BitMoD achieves an average of $1.69\times$ and $1.48\times$ speedups compared to prior LLM accelerators ANT and OliVe, respectively.

Figures (10)

• Figure 1: Total memory access of weights and activations on discriminative tasks (with 256 input tokens and 1 output token) and generative tasks (with 256 input tokens and 256 generated tokens). Note the log scale on the y-axis. Note that the gap between weight and activation memory accesses increases for generative tasks at batch size 1 despite a much larger KV-cache than discriminative tasks. While prior work flexgen has correctly reported a memory bottleneck caused by the KV-cache, this only occurs for 175B+ parameter models with a high batch size (e.g., 512) and a context lengths exceeding 512 tokens. This scenario is less relevant to our focus on low-batch edge LLM inference where the weights indeed dominate the total memory accesses.
• Figure 2: Maximum value and value range for different quantization granularity. Results are normalized to the standard deviation ($\sigma$) of the weight vector at the corresponding granularity, then averaged across all weight vectors. The per-group granularity has a group size of 128.
• Figure 3: Normalized weight quantization error ($\downarrow$) with different special values (SV) for FP3. We use per-group quantization with a group size of 128. The special values $\pm\,6$ achieve the lowest overall quantization error, thus adopted in BitMoD.
• Figure 4: Unified bit-serial representation of (a) INT8, INT6, and (b) FP4. Every bit-serial term contains four parts: sign, exponent (exp), mantissa (man) and bit-significance (bsig).
• Figure 5: The microarchitecture of BitMoD PE. Every bit-serial weight term contains 1-bit sign ($w_s$), 2-bit exponent ($w_e$), 1-bit mantissa ($w_m$), and a shared bit-significance ($w_{bsig}$).