Hummingbird: A Smaller and Faster Large Language Model Accelerator on Embedded FPGA

TL;DR

Hummingbird tackles the challenge of running large language models on embedded FPGAs by introducing a compact, DSP-efficient accelerator that supports LLaMA3-8B with extended context. Central to the design are a $INT24$ GEMV compute engine, a column-aligned memory access strategy to mitigate AXI arbitration, and an embedding offload plus Grouped-Query Attention dataflow that extends context length to $4096$ tokens. The paper demonstrates substantial improvements in resource efficiency (e.g., 67% LUT, 39% DSP, 42% power savings), throughput (up to $8.6$ tokens/s on ZCU104), and bandwidth utilization ($93$-$94\%$) on embedded FPGAs, while enabling deployment on cost-optimized FPGA platforms. These results position embedded FPGAs as practical venues for edge LLM inference, offering a favorable balance of cost, throughput, and power for real-time applications.

Abstract

Deploying large language models (LLMs) on embedded devices remains a significant research challenge due to the high computational and memory demands of LLMs and the limited hardware resources available in such environments. While embedded FPGAs have demonstrated performance and energy efficiency in traditional deep neural networks, their potential for LLM inference remains largely unexplored. Recent efforts to deploy LLMs on FPGAs have primarily relied on large, expensive cloud-grade hardware and have only shown promising results on relatively small LLMs, limiting their real-world applicability. In this work, we present Hummingbird, a novel FPGA accelerator designed specifically for LLM inference on embedded FPGAs. Hummingbird is smaller, targeting embedded FPGAs such as the KV260 and ZCU104 with 67% LUT, 39% DSP, and 42% power savings over existing research. Hummingbird is stronger, targeting LLaMA3-8B and supporting longer contexts, overcoming the typical 4GB memory constraint of embedded FPGAs through offloading strategies. Finally, Hummingbird is faste, achieving 4.8 tokens/s and 8.6 tokens/s for LLaMA3-8B on the KV260 and ZCU104 respectively, with 93-94% model bandwidth utilization, outperforming the prior 4.9 token/s for LLaMA2-7B with 84% bandwidth utilization baseline. We further demonstrate the viability of industrial applications by deploying Hummingbird on a cost-optimized Spartan UltraScale FPGA, paving the way for affordable LLM solutions at the edge.

Figures (13)

• Figure 1: Hummingbird system on two embedded FPGA platforms.
• Figure 2: Existing inefficiencies in LLM accelerator on embedded FPGA and proposed improvements.
• Figure 3: FP16 and INT4 quantized model size breakdowns show that the embedding table occupies a significantly larger proportion in quantized models, necessitating offloading strategy to free up memory space.
• Figure 4: A classic Zynq UltraScale+ SoC with key component connections; PL memory access to PS DRAM via multiple AXI ports exists bandwidth loss.
• Figure 5: Architecture overview of Hummingbird. Connections between the PS, MMU, VPU and SPU are labeled (1-11) and explained in the main text.