PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference

TL;DR

CENT tackles the memory-bound nature of decoder-only LLM inference by replacing GPUs with a GPU-free, CXL-based memory expansion and a hierarchical PIM-PNM compute fabric. It introduces three parallelism strategies—PP, TP, and a hybrid TP-PP—along with a detailed transformer-block mapping that keeps most compute near memory. Empirical results on Llama2 models show CENT achieving about $2.3\times$ higher end-to-end throughput, $2.9\times$ energy efficiency, and $5.2\times$ more tokens per dollar than GPU baselines, with even larger gains for longer contexts. The work demonstrates a scalable, generalizable path for efficient, cost-effective LLM inference without traditional GPU acceleration.

Abstract

Large Language Model (LLM) inference uses an autoregressive manner to generate one token at a time, which exhibits notably lower operational intensity compared to earlier Machine Learning (ML) models such as encoder-only transformers and Convolutional Neural Networks. At the same time, LLMs possess large parameter sizes and use key-value caches to store context information. Modern LLMs support context windows with up to 1 million tokens to generate versatile text, audio, and video content. A large key-value cache unique to each prompt requires a large memory capacity, limiting the inference batch size. Both low operational intensity and limited batch size necessitate a high memory bandwidth. However, contemporary hardware systems for ML model deployment, such as GPUs and TPUs, are primarily optimized for compute throughput. This mismatch challenges the efficient deployment of advanced LLMs and makes users pay for expensive compute resources that are poorly utilized for the memory-bound LLM inference tasks. We propose CENT, a CXL-ENabled GPU-Free sysTem for LLM inference, which harnesses CXL memory expansion capabilities to accommodate substantial LLM sizes, and utilizes near-bank processing units to deliver high memory bandwidth, eliminating the need for expensive GPUs. CENT exploits a scalable CXL network to support peer-to-peer and collective communication primitives across CXL devices. We implement various parallelism strategies to distribute LLMs across these devices. Compared to GPU baselines with maximum supported batch sizes and similar average power, CENT achieves 2.3$\times$ higher throughput and consumes 2.9$\times$ less energy. CENT enhances the Total Cost of Ownership (TCO), generating 5.2$\times$ more tokens per dollar than GPUs.

Figures (19)

• Figure 1: Llama2-70B touvron2023llamalonglora inference throughput and memory requirement on 4 A100 80GB GPUs.
• Figure 2: (a) Llama2-70B inference query latency increases with larger batches on 4 A100 80GB GPUs, Prompt size=512, Decoding size=3584. (b) GPU compute utilization, measured by Nvidia Nsight Compute profiler on 4 GPUs for Llama2-70B and 1 GPU for the other two models.
• Figure 3: (a) Prefill stage encodes prompt tokens in parallel. Decoding stage generates output tokens sequentially. (b) LLM contains N$\times$ decoder transformer blocks. (c) Llama2 model architecture.
• Figure 4: CENT Architecture.
• Figure 5: CXL Device Architecture.