HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

TL;DR

HLSTransform presents an energy-efficient FPGA-based approach for Llama 2 inference using High Level Synthesis. By applying 8-bit symmetric quantization (Q8_0), careful HLS optimizations (pipelining, unrolling, memory partitioning, burst AXI4), and a host–kernel architecture, the method achieves substantial energy reductions per token and competitive latency relative to CPU and GPU on a 110M-parameter model. Key contributions include a dense, open-source FPGA design for transformer inference, validated perplexity with minimal loss, and detailed guidance to enable broader adoption of FPGAs for energy-conscious AI deployments. The work demonstrates the feasibility and practical impact of energy-efficient transformer inference on FPGAs, offering a path toward democratizing access to FPGA-based acceleration for LLMs.

Abstract

Graphics Processing Units (GPUs) have become the leading hardware accelerator for deep learning applications and are used widely in training and inference of transformers; transformers have achieved state-of-the-art performance in many areas of machine learning and are especially used in most modern Large Language Models (LLMs). However, GPUs require large amounts of energy, which poses environmental concerns, demands high operational costs, and causes GPUs to be unsuitable for edge computing. We develop an accelerator for transformers, namely, Llama 2, an open-source state-of-the-art LLM, using high level synthesis (HLS) on Field Programmable Gate Arrays (FPGAs). HLS allows us to rapidly prototype FPGA designs without writing code at the register-transfer level (RTL). We name our method HLSTransform, and the FPGA designs we synthesize with HLS achieve up to a 12.75x reduction and 8.25x reduction in energy used per token on the Xilinx Virtex UltraScale+ VU9P FPGA compared to an Intel Xeon Broadwell E5-2686 v4 CPU and NVIDIA RTX 3090 GPU respectively, while increasing inference speeds by up to 2.46x compared to CPU and maintaining 0.53x the speed of an RTX 3090 GPU despite the GPU's 4 times higher base clock rate. With the lack of existing open-source FPGA accelerators for transformers, we open-source our code and document our steps for synthesis. We hope this work will serve as a step in democratizing the use of FPGAs in transformer inference and inspire research into energy-efficient inference methods as a whole. The code can be found on https://github.com/HLSTransform/submission.

Figures (2)

• Figure 1: Vitis HLS development workflow.
• Figure 2: Pipelining two iterations of instructions with read, execute, and write stages.