Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

TL;DR

The paper investigates FPGA-based spatial accelerators for large language model inference, addressing the latency and memory access overheads of temporal overlays. It introduces an analytical framework to model compute, memory, and bandwidth constraints on FPGAs for Transformer blocks, distinguishing prefill and decode phases, and extends to distributed multi-FPGA setups. A library of modular, reusable HLS kernels is developed to enable composable LLM accelerators, and practical designs for BERT and GPT-2 are demonstrated on an AMD Alveo U280, achieving up to 13.4x speedup over prior FPGA approaches and notable energy efficiency gains versus GPUs in decoding. The results highlight the potential of model-specific spatial acceleration on FPGAs, especially in memory-bound decode stages and multi-device configurations, while outlining remaining challenges in memory bandwidth, timing closure, and heterogeneous deployments.

Abstract

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. The majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. Through our analysis, we can determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4x speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Figures (13)

• Figure 1: Temporal and spatial architectures --- PE stands for processing engine; $f_1$-$f_4$ represent different operators in the model.
• Figure 2: Transformer model. Red blocks represent linear operators, and blue blocks signify non-linear operators.
• Figure 3: An example of tensor parallelism of a Transformer layer with two devices. TP rank is the unique identifier given to a device within a TP group. SM is the softmax function, LN is LayerNorm, and GL is the GeLU function.
• Figure 4: Pipeline diagram. Different colors stand for different input samples. Different blocks stand for different linear operators which also constitute the pipeline stages. $h$ is the number of attention heads.
• Figure 5: Blocking and non-blocking all-reduce in TP. The latency of different stages is not drawn to scale.