Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference

Seungjae Moon, Jung-Hoon Kim, Junsoo Kim, Seongmin Hong, Junseo Cha, Minsu Kim, Sukbin Lim, Gyubin Choi, Dongjin Seo, Jongho Kim, Hunjong Lee, Hyunjun Park, Ryeowook Ko, Soongyu Choi, Jongse Park, Jinwon Lee, Joo-Young Kim

TL;DR

The paper introduces HyperAccel's LPU, a latency-optimized, scalable processor for large language model inference, complemented by the Expandable Synchronization Link (ESL) and the HyperDex software framework. LPU achieves high memory bandwidth efficiency through a streamlined dataflow and tile-based execution, while ESL overlaps computation and synchronization to enable near-linear multi-device scaling. HyperDex automates model compilation and runtime integration with HuggingFace-like APIs, easing deployment across datacenters. Experimental results on ASIC and FPGA prototypes show substantial latency reductions and energy efficiency improvements over NVIDIA GPUs, with strong scalability across multiple LPUs and configurations, underscoring practical impact for GenAI workloads in cloud and edge environments.

Abstract

The explosive arrival of OpenAI's ChatGPT has fueled the globalization of large language model (LLM), which consists of billions of pretrained parameters that embodies the aspects of syntax and semantics. HyperAccel introduces latency processing unit (LPU), a latency-optimized and highly scalable processor architecture for the acceleration of LLM inference. LPU perfectly balances the memory bandwidth and compute logic with streamlined dataflow to maximize performance and efficiency. LPU is equipped with expandable synchronization link (ESL) that hides data synchronization latency between multiple LPUs. HyperDex complements LPU as an intuitive software framework to run LLM applications. LPU achieves 1.25 ms/token and 20.9 ms/token for 1.3B and 66B model, respectively, which is 2.09x and 1.37x faster than the GPU. LPU, synthesized using Samsung 4nm process, has total area of 0.824 mm2 and power consumption of 284.31 mW. LPU-based servers achieve 1.33x and 1.32x energy efficiency over NVIDIA H100 and L4 servers, respectively.

LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference

TL;DR

The paper introduces HyperAccel's LPU, a latency-optimized, scalable processor for large language model inference, complemented by the Expandable Synchronization Link (ESL) and the HyperDex software framework. LPU achieves high memory bandwidth efficiency through a streamlined dataflow and tile-based execution, while ESL overlaps computation and synchronization to enable near-linear multi-device scaling. HyperDex automates model compilation and runtime integration with HuggingFace-like APIs, easing deployment across datacenters. Experimental results on ASIC and FPGA prototypes show substantial latency reductions and energy efficiency improvements over NVIDIA GPUs, with strong scalability across multiple LPUs and configurations, underscoring practical impact for GenAI workloads in cloud and edge environments.

Abstract

The explosive arrival of OpenAI's ChatGPT has fueled the globalization of large language model (LLM), which consists of billions of pretrained parameters that embodies the aspects of syntax and semantics. HyperAccel introduces latency processing unit (LPU), a latency-optimized and highly scalable processor architecture for the acceleration of LLM inference. LPU perfectly balances the memory bandwidth and compute logic with streamlined dataflow to maximize performance and efficiency. LPU is equipped with expandable synchronization link (ESL) that hides data synchronization latency between multiple LPUs. HyperDex complements LPU as an intuitive software framework to run LLM applications. LPU achieves 1.25 ms/token and 20.9 ms/token for 1.3B and 66B model, respectively, which is 2.09x and 1.37x faster than the GPU. LPU, synthesized using Samsung 4nm process, has total area of 0.824 mm2 and power consumption of 284.31 mW. LPU-based servers achieve 1.33x and 1.32x energy efficiency over NVIDIA H100 and L4 servers, respectively.
Paper Structure (24 sections, 7 figures, 1 table)

This paper contains 24 sections, 7 figures, 1 table.

Table of Contents

  1. Background and Motivation
  2. Structure of Large Language Model
  3. Diversity of Large Language Model
  4. Inefficiency in Conventional Hardware
  5. Latency Processing Unit
  6. Hardware Architecture
  7. LPU Dataflow
  8. Instruction Set Architecture
  9. Expandable Synchronization Link
  10. Expandable Network
  11. Reconfigurable Network
  12. HyperDex Framework
  13. Compilation Layer
  14. Runtime Layer
  15. Experimental Setup
  16. ...and 9 more sections

Figures (7)

  • Figure 1: Structure of large language model.
  • Figure 2: GPU analysis when running LLM inference.
  • Figure 3: LPU hardware architecture and its dataflow.
  • Figure 4: Dataflow and timeline of data synchronization in LPU with expandable synchronization link.
  • Figure 5: HyperDex software stack for the LPU.
  • ...and 2 more figures