RACAM: Enhancing DRAM with Reuse-Aware Computation and Automated Mapping for ML Inference

TL;DR

RACAM targets memory-bandwidth bottlenecks in LLM inference by bringing bit-serialPIM inside DRAM, augmented with locality buffers, per-bank bit-serial PEs, and popcount reductions. A generalized mapping framework decouples workload structure from DRAM configuration to exhaustively search for latency-optimal GEMM mappings, while internal broadcasting reduces off-chip data movement. Empirical results show substantial end-to-end performance gains over GPUs and prior PIMs, with strong area efficiency and robustness via ablation and sensitivity analyses. The work combines architectural innovation with a practical mapping flow to unlock DRAM-scale parallelism for large-scale AI workloads. The integration of bit-level reuse, internal broadcasting, and systematic mapping yields significant throughput and energy-efficiency advantages for end-to-end LLM inference.

Abstract

In-DRAM Processing-In-Memory (DRAM-PIM) has emerged as a promising approach to accelerate memory-intensive workloads by mitigating data transfer overhead between DRAM and the host processor. Bit-serial DRAM-PIM architectures, further enhance efficiency by supporting runtime variable data precision, which is critical for emerging workloads, such as large language model (LLM) inference. However, existing works still have major limitations: lack of data reuse, significant amounts of redundant data transfer, and insufficient support for workload mapping. To address these issues, we propose RACAM, the first in-DRAM bit-serial architecture which uses dedicated locality buffers, bit-serial PEs, popcount reduction units and broadcast units to enable data reuse and alleviate redundant data transfers. Furthermore, a workload mapping mechanism is proposed to fully explore the massive parallelism of DRAM architecture and identify the best mapping scheme of a given workload. We evaluate RACAM against GPUs and the state-of-the-art, in-DRAM PIM system, Proteus, across end-to-end LLM inferences. RACAM achieves 9x to 102x speedup over GPUs and 233x higher performance per mm2 compared to Proteus in case of GPT3.

Figures (17)

• Figure 1: Integer Multiplication Latency
• Figure 2: DRAM organization overview.
• Figure 3: Tiling of Matrix Multiplication
• Figure 4: RACAM system overview. Added peripherals are colored in yellow; broadcasting units are not shown.
• Figure 5: Added peripheral units to RACAM architecture