Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing

TL;DR

This work tackles the memory bottleneck of KV caches when running LLMs on edge devices. It introduces Kelle, a software-hardware co-design that uses eDRAM as the primary KV storage and couples it with an attention-based eviction and recomputation policy (AERP) and a two-dimensional adaptive refresh policy (2DRP); a dedicated eDRAM controller, systolic evictor, and Kelle scheduler coordinate data lifetime to dramatically reduce refresh energy and memory traffic. The approach achieves up to $3.9\times$ speedup and $4.5\times$ energy savings with negligible accuracy loss, outperforming baselines and other accelerators on long-context tasks. These results demonstrate the viability of dense on-chip KV caches using eDRAM for efficient edge LLM serving and suggest a path for integrating similar co-designs with GPUs or other accelerators. The work highlights a practical, scalable route to bring private, low-latency LLM inference to edge devices without compromising accuracy or overall system efficiency.

Abstract

Running Large Language Models (LLMs) on edge devices is crucial for reducing latency, improving real-time processing, and enhancing privacy. By performing inference directly on the device, data does not need to be sent to the cloud, ensuring faster responses and reducing reliance on network connectivity. However, implementing LLMs on edge devices presents challenges, particularly with managing key-value (KV) caches, which plays a pivotal role in LLM serving. As the input text lengthens, the size of the KV cache increases linearly with the sequence length, leading to a significant memory footprint and data access costs. On the other hand, edge devices have limited memory and computational power, making it hard to store and efficiently access the large caches needed for LLM inference. To mitigate the substantial overhead caused by KV cache, we propose using embedded DRAM (eDRAM) as the primary storage for LLM serving in edge device, which offers higher storage density compared to SRAM. However, to ensure data integrity, eDRAM needs periodic refresh operations, which are power-intensive. To reduce eDRAM costs and improve overall system performance, we propose~\textit{Kelle}, a software-hardware co-design solution optimized for deploying LLMs on eDRAM-based edge systems. Combined with our fine-grained memory eviction, recomputation, and refresh control algorithms, the \textit{Kelle} accelerator delivers a $3.9\times$ speedup and $4.5\times$ energy savings compared to existing baseline solutions.

Figures (16)

• Figure 1: (a) LLM token generation. (b) KV cache for intermediate data storage, where N denotes the token index.
• Figure 2: An example on KV vector computation.
• Figure 3: (a) Normalized latency of edge systems with 4MB vs. 8MB SRAM across models and sequence lengths. (b) Area breakdown of the edge systems with 8MB eDRAM and 8MB SRAM. Red line is the area budget. (c) Energy breakdown of the edge system integrating eDRAM. The decoding lengths are shown with a prefilling length of 512. An 8MB eDRAM is used to store KV cache for a subset of layers during decoding. The reported DRAM energy accounts for both model weight access and KV cache offloading from eDRAM.
• Figure 4: 65nm eDRAM retention failure distribution at $105^{\circ}C$edram_distribute.
• Figure 5: Overview of the Kelle accelerator.