FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Design

TL;DR

FlexLLM introduces a composable high-level synthesis library that enables stage-customized hybrid architectures for LLM accelerators, addressing the divergent bottlenecks of prefill and decode while providing a comprehensive, hardware-aware quantization stack. It demonstrates rapid construction of Llama-3.2 1B accelerators with SpinQuant and shows substantial gains in end-to-end speed, decode throughput, and energy efficiency compared to a GPU baseline. The framework is extended with a Hierarchical Memory Transformer plug-in to enable long-context processing, delivering large reductions in prefill latency and dramatic context window expansion with modest hardware overhead. Together, these contributions bridge algorithmic innovation and hardware design, enabling model-to-silicon turnaround in weeks and scalable deployment across edge to data-center platforms.

Abstract

We present FlexLLM, a composable High-Level Synthesis (HLS) library for rapid development of domain-specific LLM accelerators. FlexLLM exposes key architectural degrees of freedom for stage-customized inference, enabling hybrid designs that tailor temporal reuse and spatial dataflow differently for prefill and decode, and provides a comprehensive quantization suite to support accurate low-bit deployment. Using FlexLLM, we build a complete inference system for the Llama-3.2 1B model in under two months with only 1K lines of code. The system includes: (1) a stage-customized accelerator with hardware-efficient quantization (12.68 WikiText-2 PPL) surpassing SpinQuant baseline, and (2) a Hierarchical Memory Transformer (HMT) plug-in for efficient long-context processing. On the AMD U280 FPGA at 16nm, the accelerator achieves 1.29$\times$ end-to-end speedup, 1.64$\times$ higher decode throughput, and 3.14$\times$ better energy efficiency than an NVIDIA A100 GPU (7nm) running BF16 inference; projected results on the V80 FPGA at 7nm reach 4.71$\times$, 6.55$\times$, and 4.13$\times$, respectively. In long-context scenarios, integrating the HMT plug-in reduces prefill latency by 23.23$\times$ and extends the context window by 64$\times$, delivering 1.10$\times$/4.86$\times$ lower end-to-end latency and 5.21$\times$/6.27$\times$ higher energy efficiency on the U280/V80 compared to the A100 baseline. FlexLLM thus bridges algorithmic innovation in LLM inference and high-performance accelerators with minimal manual effort.

Figures (8)

• Figure 1: Comparison of three architectural styles for LLM accelerators across prefill and decode stages. Different colors indicate different tokens and blocks denote hardware modules. Only the linear layer is shown for clarity, where $A$ denotes the Multi-Head Attention (Grouped-Query Attention here) and $O$ denotes the output projection. (a) One Transformer block. (b–c) Temporal architectures achieve high utilization via module reuse but suffer from frequent off-chip memory access and limited flexibility. (d–e) Spatial architectures dedicate modules per kernel for full on-chip streaming but are sensitive to pipeline stalls. (f–g) Hybrid architectures combine temporal reuse and spatial parallelism to balance utilization, latency, and flexibility.
• Figure 2: Profiling of compute throughput and memory bandwidth utilization during the prefill and decode stages of the BF16 Llama-3.2 1B model on an NVIDIA A100 GPU.
• Figure 3: stage-customized module design for (a) prefill, (b) decode, and (c) quantization module integration.
• Figure 4: Example code illustrating hybrid architecture construction combining temporal reuse and spatial dataflow with FlexLLM.
• Figure 5: Hybrid architecture design for (a) prefill & (b) decode, and (c) the integration diagram of HMT plug-in.