Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

Zicheng He, Anhao Zhao, Xiaoyu Shen, Chen Wu, Lei He

Abstract

Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their inference efficiency remains a critical bottleneck due to rapidly growing parameters. Recent advances in dynamic computation allocation address this challenge by exploiting the highly uneven contributions of different tokens and layers, enabling selective execution that significantly reduces redundant computation while preserving model accuracy. However, existing hardware platforms and accelerators are primarily optimized for uniform, static execution, limiting their ability to efficiently support such dynamic inference patterns. In this work, we propose SkipOPU, an FPGA-based overlay processor that dynamically allocates computation across tokens and layers with high flexibility through a lightweight routing mechanism. First, we decouple reduction operations from element-wise computation in nonlinear modules and perform reductions incrementally, which enables both stages to be fused with adjacent linear operations (router or matrix multiplication) for effective latency hiding. Second, motivated by asymmetric sensitivity to numerical precision between activation and weight, we design a PE array that efficiently supports float-fixed hybrid execution. A novel DSP overpacking technique is introduced to maximize hardware utilization while minimizing resource overhead. Finally, we develop a proactive on-chip KV history buffer that exploits cross-layer KV invariance of pruned tokens, eliminating irregular HBM accesses during decoding and supplementing off-chip bandwidth through high-locality on-chip reuse. Experimental results demonstrate that SkipOPU on an AMD U280 FPGA outperforms GPU and other FPGA-based accelerators by 1.23x-3.83x in bandwidth efficiency for LLMs inference with dynamic computation allocation and can reduce up to 25.4% KV storage overhead across varying sequence lengths.

SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

Abstract

Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their inference efficiency remains a critical bottleneck due to rapidly growing parameters. Recent advances in dynamic computation allocation address this challenge by exploiting the highly uneven contributions of different tokens and layers, enabling selective execution that significantly reduces redundant computation while preserving model accuracy. However, existing hardware platforms and accelerators are primarily optimized for uniform, static execution, limiting their ability to efficiently support such dynamic inference patterns. In this work, we propose SkipOPU, an FPGA-based overlay processor that dynamically allocates computation across tokens and layers with high flexibility through a lightweight routing mechanism. First, we decouple reduction operations from element-wise computation in nonlinear modules and perform reductions incrementally, which enables both stages to be fused with adjacent linear operations (router or matrix multiplication) for effective latency hiding. Second, motivated by asymmetric sensitivity to numerical precision between activation and weight, we design a PE array that efficiently supports float-fixed hybrid execution. A novel DSP overpacking technique is introduced to maximize hardware utilization while minimizing resource overhead. Finally, we develop a proactive on-chip KV history buffer that exploits cross-layer KV invariance of pruned tokens, eliminating irregular HBM accesses during decoding and supplementing off-chip bandwidth through high-locality on-chip reuse. Experimental results demonstrate that SkipOPU on an AMD U280 FPGA outperforms GPU and other FPGA-based accelerators by 1.23x-3.83x in bandwidth efficiency for LLMs inference with dynamic computation allocation and can reduce up to 25.4% KV storage overhead across varying sequence lengths.
Paper Structure (29 sections, 2 equations, 9 figures, 3 tables, 2 algorithms)

This paper contains 29 sections, 2 equations, 9 figures, 3 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND AND MOTIVATIONS
  3. SkipGPT framework
  4. Motivation
  5. DATAFLOW OPTIMIZATION
  6. Fused dataflow for routing and RMSNorm
  7. Fused dataflow for self-attention mechanism
  8. HARDWARE ARCHITECTURE
  9. Overview
  10. Mixed-precision PE array
  11. Overpacking design for FP16 with truncation
  12. Mixed-precision PE architecture
  13. Validation
  14. Tile-based nonlinear processing engine
  15. Optimization with cross-layer KV reuse
  16. ...and 14 more sections

Figures (9)

  • Figure 1: (a) The framework of SkipGPT models. (b) Challenges in SkipGPT computation, and the corresponding solution in SkipOPU.
  • Figure 2: Overview of SkipOPU where HBM operates at high clock domain and accelerator operates at low clock domain
  • Figure 3: Comparison between differences DSP packing techniques for FP16 fraction multiplication.
  • Figure 4: (a) The detailed design of mixed precision PE. (b) FP16 accumulation tree with BFP-FP convertion
  • Figure 5: The micro-architecture of nonlinear processing engine(NPE).
  • ...and 4 more figures