CD-PIM: A High-Bandwidth and Compute-Efficient LPDDR5-Based PIM for Low-Batch LLM Acceleration on Edge-Device

TL;DR

Edge-hosted LLM decode is bandwidth-bound, while compute resources are underutilized in existing PIM designs. CD-PIM addresses this by partitioning every LPDDR5 bank into four Pbanks to boost internal bandwidth and by employing two operation modes: HBCEM to maximize GEMV throughput and LBIM to overlap GEMV with GEMM, enabling concurrent PIM and host processing. A compute-efficient, pipelined CU supports both inner- and outer-product operations with carefully designed K-cache and V-cache mappings, achieving substantial speedups over GPU and SOTA PIM baselines ($11.42×$–$14.6×$ in HBCEM; $1.01×$–$1.46×$ LBIM for low-batch). The approach demonstrates favorable area and power overhead ($0.8 ext{%}$ of a LPDDR5 die, ~144 mW total) while delivering practical gains for edge-LM inference with low batch sizes and memory-intensive decoding, enabling real-time on-device capabilities.

Abstract

Edge deployment of low-batch large language models (LLMs) faces critical memory bandwidth bottlenecks when executing memory-intensive general matrix-vector multiplications (GEMV) operations. While digital processing-in-memory (PIM) architectures promise to accelerate GEMV operations, existing PIM-equipped edge devices still suffer from three key limitations: limited bandwidth improvement, component under-utilization in mixed workloads, and low compute capacity of computing units (CUs). In this paper, we propose CD-PIM to address these challenges through three key innovations. First, we introduce a high-bandwidth compute-efficient mode (HBCEM) that enhances bandwidth by dividing each bank into four pseudo-banks through segmented global bitlines. Second, we propose a low-batch interleaving mode (LBIM) to improve component utilization by overlapping GEMV operations with GEMM operations. Third, we design a compute-efficient CU that performs enhanced GEMV operations in a pipelined manner by serially feeding weight data into the computing core. Forth, we adopt a column-wise mapping for the key-cache matrix and row-wise mapping for the value-cache matrix, which fully utilizes CU resources. Our evaluation shows that compared to a GPU-only baseline and state-of-the-art PIM designs, our CD-PIM achieves 11.42x and 4.25x speedup on average within a single batch in HBCEM mode, respectively. Moreover, for low-batch sizes, the CD-PIM achieves an average speedup of 1.12x in LBIM compared to HBCEM.

Figures (5)

• Figure 1: Detailed data mapping of the CD-PIM: (a) K-cache matrix data mapping and outer-product compute flow; (b) V-cache matrix data mapping and inner-product compute flow.
• Figure 2: Normalized performance of LLaMA-1B, -7B, and -13B (batch size $=1$) under various Lin and Lout on the NVIDIA Jetson AGX Orin and Apple iPhone 15 Pro.
• Figure 3: Normalized performance of the NVIDIA Jetson AGX Orin equipped with CD‑PIM under HBCEM and LBIM (batch size $=4$).
• Figure 4: Normalized performance of the Apple iPhone 15 Pro equipped with CD‑PIM under HBCEM and LBIM (batch size $=4$).
• Figure 5: The area and power breakdown of CU.