Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs

Ye Qiao, Zhiheng Chen, Yifan Zhang, Yian Wang, Sitao Huang

TL;DR

TeLLMe tackles the challenge of end-to-end LLM inference on edge FPGAs by co-designing a table-lookup-based ternary MatMul engine with URAM-aware buffering and streaming dataflow to support both prefill and decoding. The core ideas include a TLMM that uses online precomputation and grouped activations, fused element-wise operations around mixed-precision paths, and specialized attention units (reversed prefill and decoding) that minimize off-chip traffic. Experimental results on a KV260 platform show up to $143$ tokens/s for prefill and $25$ tokens/s for decoding under a $5$ W budget, with favorable energy efficiency metrics such as Intelligence/J. The significance lies in delivering end-to-end edge inference for ternary LLMs with strong energy efficiency, enabling practical privacy-preserving generative AI on small devices. The work establishes a strong baseline for implementing low-bit quantized LLMs on edge FPGAs via LUT-based TLMM and memory-hierarchy optimizations.

Abstract

With the emergence of wearable devices and other embedded systems, deploying large language models (LLMs) on edge platforms has become an urgent need. However, this is challenging because of their high computational and memory demands. Although recent low-bit quantization methods (e.g., BitNet, DeepSeek) compress weights to as low as 1.58~bits with minimal accuracy loss, edge deployment is still constrained by limited on-chip resources, power budgets, and the often-neglected long latency of the prefill stage. We present \textbf{TeLLMe}, the first table-lookup-based ternary LLM accelerator for low-power edge FPGAs that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. TeLLMe incorporates several novel techniques, including (1) a table-lookup-based ternary matrix multiplication (TLMM) engine utilizing grouped activations and online precomputation for low resource utilization and high throughput; (2) a fine-grained analytic URAM-based weight buffer management scheme for efficient loading and compute engine access; (3) a streaming dataflow architecture that fuses floating-point element-wise operations with linear computations to hide latency; (4) a reversed-reordered prefill stage attention with fused attention operations for high memory efficiency; and (5) a resource-efficient specialized decoding stage attention. Under a 5~W power budget, TeLLMe delivers up to 25~tokens/s decoding throughput and 0.45--0.96~s time-to-first-token (TTFT) for 64--128 token prompts, marking a significant energy-efficiency advancement in LLM inference on edge FPGAs.

TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs

TL;DR

TeLLMe tackles the challenge of end-to-end LLM inference on edge FPGAs by co-designing a table-lookup-based ternary MatMul engine with URAM-aware buffering and streaming dataflow to support both prefill and decoding. The core ideas include a TLMM that uses online precomputation and grouped activations, fused element-wise operations around mixed-precision paths, and specialized attention units (reversed prefill and decoding) that minimize off-chip traffic. Experimental results on a KV260 platform show up to tokens/s for prefill and tokens/s for decoding under a W budget, with favorable energy efficiency metrics such as Intelligence/J. The significance lies in delivering end-to-end edge inference for ternary LLMs with strong energy efficiency, enabling practical privacy-preserving generative AI on small devices. The work establishes a strong baseline for implementing low-bit quantized LLMs on edge FPGAs via LUT-based TLMM and memory-hierarchy optimizations.

Abstract

With the emergence of wearable devices and other embedded systems, deploying large language models (LLMs) on edge platforms has become an urgent need. However, this is challenging because of their high computational and memory demands. Although recent low-bit quantization methods (e.g., BitNet, DeepSeek) compress weights to as low as 1.58~bits with minimal accuracy loss, edge deployment is still constrained by limited on-chip resources, power budgets, and the often-neglected long latency of the prefill stage. We present \textbf{TeLLMe}, the first table-lookup-based ternary LLM accelerator for low-power edge FPGAs that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. TeLLMe incorporates several novel techniques, including (1) a table-lookup-based ternary matrix multiplication (TLMM) engine utilizing grouped activations and online precomputation for low resource utilization and high throughput; (2) a fine-grained analytic URAM-based weight buffer management scheme for efficient loading and compute engine access; (3) a streaming dataflow architecture that fuses floating-point element-wise operations with linear computations to hide latency; (4) a reversed-reordered prefill stage attention with fused attention operations for high memory efficiency; and (5) a resource-efficient specialized decoding stage attention. Under a 5~W power budget, TeLLMe delivers up to 25~tokens/s decoding throughput and 0.45--0.96~s time-to-first-token (TTFT) for 64--128 token prompts, marking a significant energy-efficiency advancement in LLM inference on edge FPGAs.
Paper Structure (33 sections, 11 equations, 11 figures, 4 tables)

This paper contains 33 sections, 11 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Transformer/LLM Acceleration on Edge
  4. LLMs on High-performance FPGAs
  5. Existing Table Lookup Works on FPGAs
  6. TeLLMe Design Methodology
  7. Overview
  8. Table-Lookup Matmul (TLMM) Engine
  9. TLMM Engine Design
  10. Comparison between Different Ternary Matrix Multiplication Designs
  11. Fused Element-wise Operations
  12. INT8 Quantization and FP16 Dequantization
  13. Elementwise Activation-fused Multiplication and Addition
  14. RoPE Operation
  15. Weight Buffer Management Unit (WBMU)
  16. ...and 18 more sections

Figures (11)

  • Figure 1: Breakdown of TeLLMe 1.58-bit Model Inference Process with Prefill and Generation
  • Figure 2: System Architecture of TeLLMe
  • Figure 3: High Level Dataflow of TLMM ($G = 3, T=3,Q=3$)
  • Figure 4: Overview of TLMM hardware components and dataflow. (a) The TLMM-FUSE unit design. (b) The comparison of different RoPE operations. (c) The dataflow execution schedule.
  • Figure 5: The WBMU and AXI data packing strategy. The green arrows denote the connection to the weight access, and the yellow arrows are connected to the weight load module.
  • ...and 6 more figures