Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format

TL;DR

Anda introduces a variable-length grouped activation data format with a shared exponent to accelerate FP-INT GeMM in weight-only quantized LLMs. An offline adaptive precision search selects per-module mantissa lengths via a one-shot calibration, balancing accuracy with speed and energy; a hardware splitter comprising a bit-plane memory layout, Anda bit-serial processing units, and a runtime bit-plane compressor enables efficient execution. Across OPT, LLaMA, and LLaMA-2 models, Anda achieves approximately 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency gains over a GPU-like FP-FP baseline, with targeted accuracy losses under 1% in most cases. The approach reduces FP activation data movement bottlenecks and offers adaptable deployment across diverse application scenarios, hardware platforms, and accuracy requirements.

Abstract

The widely-used, weight-only quantized large language models (LLMs), which leverage low-bit integer (INT) weights and retain floating-point (FP) activations, reduce storage requirements while maintaining accuracy. However, this shifts the energy and latency bottlenecks towards the FP activations that are associated with costly memory accesses and computations. Existing LLM accelerators focus primarily on computation optimizations, overlooking the potential of jointly optimizing FP computations and data movement, particularly for the dominant FP-INT GeMM operations in LLM inference. To address these challenges, we investigate the sensitivity of activation precision across various LLM modules and its impact on overall model accuracy. Based on our findings, we first propose the Anda data type: an adaptive data format with group-shared exponent bits and dynamic mantissa bit allocation. Secondly, we develop an iterative post-training adaptive precision search algorithm that optimizes the bit-width for different LLM modules to balance model accuracy, energy efficiency, and inference speed. Lastly, a suite of hardware optimization techniques is proposed to maximally exploit the benefits of the Anda format. These include a bit-plane-based data organization scheme, Anda-enhanced processing units with bit-serial computation, and a runtime bit-plane Anda compressor to simultaneously optimize storage, computation, and memory footprints. Our evaluations on FPINT GeMM operations show that Anda achieves a 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency improvement on average for popular LLMs including OPT, LLaMA, and LLaMA-2 series over the GPU-like FP-FP baseline. Anda demonstrates strong adaptability across various application scenarios, accuracy requirements, and system performance, enabling efficient LLM inference across a wide range of deployment scenarios.

Figures (18)

• Figure 1: Overview of the drop-in replacement for FP activations using the variable-length grouped Anda data type via a one-shot offline calibration process. This enables online variable-precision LLM inference, significantly improving speed and energy efficiency through the adaptive precision combination search algorithm and the Anda-aware architecture.
• Figure 2: Proportion of FP-INT GeMM operations in weight-only quantized LLMs across varying model sizes and context lengths for text generation tasks. FP-INT GeMMs dominate (> 90%) in prevalent sub-4K token applications and remain significant for 10K+ sequences.
• Figure 3: Illustration of the architecture for a weight-only quantized LLM model.
• Figure 4: The process of converting a set of FP16 numbers into different BFP numbers. BFP format is regulated by two key parameters: group size (GS) and mantissa length (M).
• Figure 5: LLM sensitivity to BFP group size (GS) and preserved mantissa bits.