Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference

Christopher Wolters, Xiaoxuan Yang, Ulf Schlichtmann, Toyotaro Suzumura

TL;DR

The paper surveys compute-in-memory (CIM) architectures as a route to accelerate transformer-based large language model inference by reducing data movement. It covers transformer foundations, traditional acceleration methods, CIM device technologies (CMOS, ReRAM, PCM, FeFET, MRAM), and a spectrum of CIM design strategies—from algorithmic enhancements to heterogeneous, full-circuit implementations and analog AI chips. It analyzes design and reliability challenges (analog non-idealities, peripheral overhead, precision, endurance) and proposes a taxonomy of strategies (hardware-aware training, resilience, high-precision techniques, and co-design). The work emphasizes hardware-software co-design, manufacturing advances, and error correction as critical for practical deployment, arguing CIM can substantially improve latency and energy efficiency for increasingly large and capable LLMs.

Abstract

Large language models (LLMs) have recently transformed natural language processing, enabling machines to generate human-like text and engage in meaningful conversations. This development necessitates speed, efficiency, and accessibility in LLM inference as the computational and memory requirements of these systems grow exponentially. Meanwhile, advancements in computing and memory capabilities are lagging behind, exacerbated by the discontinuation of Moore's law. With LLMs exceeding the capacity of single GPUs, they require complex, expert-level configurations for parallel processing. Memory accesses become significantly more expensive than computation, posing a challenge for efficient scaling, known as the memory wall. Here, compute-in-memory (CIM) technologies offer a promising solution for accelerating AI inference by directly performing analog computations in memory, potentially reducing latency and power consumption. By closely integrating memory and compute elements, CIM eliminates the von Neumann bottleneck, reducing data movement and improving energy efficiency. This survey paper provides an overview and analysis of transformer-based models, reviewing various CIM architectures and exploring how they can address the imminent challenges of modern AI computing systems. We discuss transformer-related operators and their hardware acceleration schemes and highlight challenges, trends, and insights in corresponding CIM designs.

Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference

TL;DR

The paper surveys compute-in-memory (CIM) architectures as a route to accelerate transformer-based large language model inference by reducing data movement. It covers transformer foundations, traditional acceleration methods, CIM device technologies (CMOS, ReRAM, PCM, FeFET, MRAM), and a spectrum of CIM design strategies—from algorithmic enhancements to heterogeneous, full-circuit implementations and analog AI chips. It analyzes design and reliability challenges (analog non-idealities, peripheral overhead, precision, endurance) and proposes a taxonomy of strategies (hardware-aware training, resilience, high-precision techniques, and co-design). The work emphasizes hardware-software co-design, manufacturing advances, and error correction as critical for practical deployment, arguing CIM can substantially improve latency and energy efficiency for increasingly large and capable LLMs.

Abstract

Large language models (LLMs) have recently transformed natural language processing, enabling machines to generate human-like text and engage in meaningful conversations. This development necessitates speed, efficiency, and accessibility in LLM inference as the computational and memory requirements of these systems grow exponentially. Meanwhile, advancements in computing and memory capabilities are lagging behind, exacerbated by the discontinuation of Moore's law. With LLMs exceeding the capacity of single GPUs, they require complex, expert-level configurations for parallel processing. Memory accesses become significantly more expensive than computation, posing a challenge for efficient scaling, known as the memory wall. Here, compute-in-memory (CIM) technologies offer a promising solution for accelerating AI inference by directly performing analog computations in memory, potentially reducing latency and power consumption. By closely integrating memory and compute elements, CIM eliminates the von Neumann bottleneck, reducing data movement and improving energy efficiency. This survey paper provides an overview and analysis of transformer-based models, reviewing various CIM architectures and exploring how they can address the imminent challenges of modern AI computing systems. We discuss transformer-related operators and their hardware acceleration schemes and highlight challenges, trends, and insights in corresponding CIM designs.
Paper Structure (38 sections, 6 equations, 6 figures, 3 tables)

This paper contains 38 sections, 6 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Advancements in Large Language Models
  3. Scaling Challenges in Neural Networks
  4. Efficiency Gains through Compute-in-Memory
  5. Hardware-Software Co-Design in CIM Systems
  6. Theoretical Background
  7. Transformer Architecture
  8. Attention Computation
  9. Large Language Models
  10. Conventional Approaches to LLM Acceleration
  11. Compute-in-Memory
  12. Memory Device Technologies
  13. CMOS
  14. ReRAM
  15. PCM
  16. ...and 23 more sections

Figures (6)

  • Figure 1: Model size of state-of-the-art LLMs 39
  • Figure 2: The transformer model architecture att
  • Figure 3: Scaled dot-product attention and multi-head attention att
  • Figure 4: Distribution of execution times of the different operations in transformer networks at varying sequence lengths 38
  • Figure 5: Memristor crossbar architecture crossbar
  • ...and 1 more figures