PRIMAL: Processing-In-Memory Based Low-Rank Adaptation for LLM Inference Accelerator

TL;DR

PRIMAL addresses the rising energy and throughput demands of LoRA-enabled LLM inference by exploiting processing-in-memory with a heterogeneous PE design and a 2D-mesh IPCN. It introduces SRPG to pipeline SRAM-DCIM reprogramming with computation and uses layer-wise CT allocation to exploit sequential execution, achieving sub-linear power scaling. It reports a $1.5\times$ throughput and $25\times$ energy efficiency improvement over Nvidia H100 for Llama-13B with LoRA rank 8 (Q,V) and demonstrates scalability through detailed hardware/software co-design and benchmarking. This work suggests PRIMAL as a scalable, energy-efficient path for deploying larger LoRA-enabled LLMs in resource-constrained settings.

Abstract

This paper presents PRIMAL, a processing-in-memory (PIM) based large language model (LLM) inference accelerator with low-rank adaptation (LoRA). PRIMAL integrates heterogeneous PIM processing elements (PEs), interconnected by 2D-mesh inter-PE computational network (IPCN). A novel SRAM reprogramming and power gating (SRPG) scheme enables pipelined LoRA updates and sub-linear power scaling by overlapping reconfiguration with computation and gating idle resources. PRIMAL employs optimized spatial mapping and dataflow orchestration to minimize communication overhead, and achieves $1.5\times$ throughput and $25\times$ energy efficiency over NVIDIA H100 with LoRA rank 8 (Q,V) on Llama-13B.

Figures (6)

• Figure 1: The architecture of LoRA.
• Figure 2: Design of PRIMAL hardware (per compute tile): heterogeneous PE, 2D-Mesh IPCN, computational router.
• Figure 3: The network main controller (NMC).
• Figure 4: The spatial mapping of weight matrices of an attention layer.
• Figure 5: Our Proposed SRAM Reprogramming and Power Gating (SRPG) scheme.