AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design

TL;DR

AccLLM addresses the critical challenge of deploying long-context LLMs on edge devices by jointly optimizing algorithms and hardware. It combines 2:4 semi-structured pruning, Lambda-shaped attention, and W2A8KV4 quantization with an FPGA-based reconfigurable accelerator to dramatically reduce compute, memory, and KV-cache bandwidth during decode. Extensive experiments on Llama-2-7B show up to 4.07x energy efficiency and 2.98x throughput improvements over FlightLLM, with substantial KV-cache memory reductions and robust performance on long sequences. This engineering-centric co-design enables practical edge deployment of capable LLMs with high efficiency and scalability.

Abstract

Recently, large language models (LLMs) have achieved huge success in the natural language processing (NLP) field, driving a growing demand to extend their deployment from the cloud to edge devices. However, deploying LLMs on resource-constrained edge devices poses significant challenges, including (1) intensive computations and huge model sizes, (2) great memory and bandwidth demands introduced by the autoregressive generation process, and (3) limited scalability for handling long sequences. To address these challenges, we propose AccLLM, a comprehensive acceleration framework that enables efficient and fast long-context LLM inference through algorithm and hardware co-design. At the algorithmic level, we integrate (1) pruning, (2) Λ-shaped attention, and (3) an innovative W2A8KV4 (2-bit weights, 8-bit activations, and 4-bit KV cache) quantization scheme, thus effectively reducing memory and bandwidth requirements while facilitating LLMs' long-sequence generation. At the hardware level, we design a dedicated FPGA-based accelerator with a reconfigurable computing engine to effectively and flexibly accommodate diverse operations arising from our compression algorithm, thereby fully translating the algorithmic innovations into tangible hardware efficiency. We validate AccLLM on the Xilinx Alveo U280 FPGA, demonstrating a 4.07x energy efficiency and a 2.98x throughput compared to the state-of-the-art work FlightLLM.

Figures (15)

• Figure 1: The inference pipelines of LLMs (a) without KV cache and (b) with KV cache.
• Figure 2: Driven by the (1) intensive computations and large model sizes of LLMs, (2) substantial memory overhead and bandwidth demands introduced by the autoregressive generation process, and (3) limited scalability for handling long sequences, we propose a comprehensive LLM acceleration framework dubbed AccLLM, which incorporates (1) an aggressive compression algorithm and (2) a dedicated accelerator, thus facilitating extensive real-world applications of LLMs on edge devices.
• Figure 3: (a) The structure of LLMs. (b) Illustrating the key computations during the prefill and decode stages of LLM inference.
• Figure 4: The computational breakdown of Llama-2-7B Touvron2023Llama2O inference during (a) the prefill stage and (b) the decode stage across different sequence lengths.
• Figure 5: Roofline analysis on the Xilinx Alveo U280 FPGA for the three primary types of linear operations during Llama-2-7B Touvron2023Llama2O inference.