Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

Zhongkai Yu, Shengwen Liang, Tianyun Ma, Yunke Cai, Ziyuan Nan, Di Huang, Xinkai Song, Yifan Hao, Jie Zhang, Tian Zhi, Yongwei Zhao, Zidong Du, Xing Hu, Qi Guo, Tianshi Chen

TL;DR

Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs, and enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, showing the potentiality of deploying powerful LLMs in edge devices.

Abstract

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses the significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash's on-die processing capabilities. Overall, Cambricon-LLM enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, and 7B LLMs at a speed of 36.34 token/s, which is over 22X to 45X faster than existing flash-offloading technologies, showing the potentiality of deploying powerful LLMs in edge devices.

Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

TL;DR

Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs, and enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, showing the potentiality of deploying powerful LLMs in edge devices.

Abstract

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses the significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-LLM, a chiplet-based hybrid architecture with NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash's on-die processing capabilities. Overall, Cambricon-LLM enables the on-device inference of 70B LLMs at a speed of 3.44 token/s, and 7B LLMs at a speed of 36.34 token/s, which is over 22X to 45X faster than existing flash-offloading technologies, showing the potentiality of deploying powerful LLMs in edge devices.
Paper Structure (31 sections, 9 equations, 16 figures, 5 tables)

This paper contains 31 sections, 9 equations, 16 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Inference
  4. In-storage Computing Based on Flash
  5. Motivation
  6. Low Arithmetic Intensity of Single-batch LLM Inference
  7. Why Flash?
  8. The Necessity of Error Correction
  9. Cambricon-LLM Architecture
  10. Overview
  11. Design of Flash Die
  12. Design of Slice Control
  13. Hardware-aware Tiling
  14. Identify the Optimal Shape of Tiles
  15. Determine Optimal Workload Distribution Proportions
  16. ...and 16 more sections

Figures (16)

  • Figure 1: (a) Arithmetic Intensity comparison between LLM and other AI algorithms and (b) Reduction Ratio comparison between the scenario in LLMFlash and other ISC works.
  • Figure 2: Architecture of Flash Memory.
  • Figure 3: (a) Roofline model analysis of our architecture and smartphone NPU and (b) the sensitivity test of OPT-6.7B under bit-flip error in flash.
  • Figure 4: Overall architecture of Cambricon-LLM.
  • Figure 5: Compute flow and hardware mapping of an LLM inference layer.
  • ...and 11 more figures