Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

TL;DR

Harmonia tackles the memory and compute bottlenecks of large language model inference by extending the block floating point (BFP) format to all activations, including linear and attention layers, and by aggressively compressing the KV cache. It introduces a unified algorithm-hardware co-design: (i) a dynamic BFP conversion framework with per-token grouping and an incremental V-group strategy, (ii) an asymmetric bit allocation and hybrid outlier smoothing to preserve accuracy during KV-cache compression, and (iii) a reconfigurable PE-based hardware core with mixed-precision BFP-INT and BFP-BFP MACs and a real-time BFP converter, plus tiling-aware dataflow. The result is unified, memory- and compute-efficient LLM inference demonstrated across eight models, achieving up to 3.84×–5.05× improvements in area, energy, and speed on average, with only about $0.3\%$ average accuracy loss under aggressive KV-cache compression to $4$-bit mantissas. These advances enable scalable, long-context LLM deployment with significantly reduced hardware costs and powered by all-layer BFP activations.

Abstract

Large Language Models (LLMs) are powerful but incur high memory and computation costs. Quantization is an effective solution, with INT weights and FP activations being widely adopted to preserve accuracy. Prior works further reduce FP overhead by using block floating point (BFP) activations in linear layers, but fail to extend BFP to attention layers due to severe accuracy degradation, limiting overall efficiency. To address this challenge, we propose Harmonia, an algorithm-hardware co-design framework that enables all-layer BFP activations with a configurable hardware architecture. First, we systematically explore BFP configurations to achieve a better trade-off between accuracy and activation compression across all layers. Second, to reduce KV-cache storage and computation in attention layers, we introduce an asymmetric bit-allocation strategy and computations in attention layers,we introduce an asymmetric bit-allocation strategy combined with a hybrid offline-online outlier smoothing technique. This allow aggressive KV-cache compression from FP16 to 4-bit-mantissa BFP with only 0.3% average accuracy loss. Third, to fully exploit all-layer BFP activations, we design dedicated hardware components, including a reconfigurable PE supporting mixed data formats (BFP-INT and BPF-BFP), a real-time FP16-to-BFP converter, and a tiling-aware dataflow to reduce memory traffic. We evaluate Harmonia on GEMM operations in both linear and attention layers across eight widely used LLMs. Compared with prior works, Harmonia achieves 3.84x (up to 5.05x) higher area efficiency, 2.03x (up to 3.90x) better energy efficiency, and 3.08x (up to 4.62x) speedup on average.

Figures (19)

• Figure 1: Normalized breakdown of (a) operations and (b) memory storage for models of different scales under varying sequence lengths.
• Figure 2: Illustration of auto-regressive LLM architecture.
• Figure 3: Illustration of the FP16-to-BFP conversion with a group size of 3 and an 8-bit mantissa.
• Figure 4: Relative accuracy of different models under various preserved mantissa bits and group sizes.
• Figure 5: Relative accuracy of different models under various preserved mantissa bits of the KV cache.