SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding

TL;DR

The paper tackles edge LLM decoding bottlenecks by introducing SwiftKV Attention, a per-token, single-pass attention algorithm, and SwiftKV-MHA, a decoder-optimized multi-head accelerator. SwiftKV Attention processes each KV cache entry exactly once and defers normalization, implemented on fixed-point FXP32 with a LUT-based exponent for 2^{f}, yielding a final output $\mathrm{Attention} = \dfrac{Y_T}{Z_T}$ without intermediate score materialization. SwiftKV-MHA provides 32 independent processing units for per-head attention and supports high-throughput low-precision GEMV on the same array, augmented by decoder-special RoPE to minimize data movement. Empirically, the approach achieves a 7.16× speedup over native attention, a 13.48× reduction in attention latency, and a 17.4% increase in generation speed with 1.98× higher efficiency, validated on edge FPGA hardware with LLaMA2-7B and ChatGLM-6B models, indicating strong practical impact for edge-based LLM inference.

Abstract

Edge acceleration for large language models is crucial for their widespread application; however, achieving fast attention inference and efficient decoding on resource-constrained edge accelerators remains challenging. This paper presents SwiftKV Attention, a per-token pipelined, low-latency single-pass attention inference algorithm, where every (kt, vt) in the KV cache is processed exactly once in a uniform per-token pipeline without score materialization, blockwise softmax, or a second pass, thereby enabling fast execution on edge accelerators with a single hardware set and no resource-intensive parallelism. Furthermore, to address the limited support for multi-head LLM decoding in existing accelerators, we design the SwiftKV-MHA accelerator, which enables high precision attention and low precision GEMV on the same processor array, achieving fast and efficient multi-head parallel decoding. Experimental results show that, on the edge accelerator, the SwiftKV Attention algorithm achieves a 7.16* speedup over native attention and significantly outperforms other attention algorithms. SwiftKV-MHA further reduces attention latency by 13.48*; under the same settings, it improves generation speed by 17.4% and increases token efficiency by 1.98* compared with state-of-the-art works.

Figures (8)

• Figure 1: The structure of the multi-head LLM layer during decoding.
• Figure 2: Pipelined computing flow of SwiftKV Attention for edge accelerators.
• Figure 3: Hardware Compute Engine of SwiftKV-Attention: SwiftKV Core.
• Figure 4: Overall architecture of SwiftKV-MHA accelerator.
• Figure 5: Illustration of fast GEMV and attention computation. (a) Using multi-heads to compute per-head attention and high dimensional GEMV. (b) Sharing the same compute resources between high precision FXP32 attention and low precision multi-path GEMV. (c) Precision conversion during computation.