DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones

TL;DR

DynaKV tackles the bottleneck of long-context LLM decoding on smartphones by introducing adaptive KVCache management that responds to KVCache distribution shifts during decoding. It combines three techniques—migration-free cluster adaptation, continuity-centric flash management, and memory-efficient cache design—to jointly optimize retrieval accuracy and end-to-end latency. Empirical results show average accuracy gains of 1.38x and end-to-end latency speedups of 1.47x, along with improved effective bandwidth across diverse models and devices. The approach is applicable to other long-context workloads and broader multi-tier memory hierarchies, highlighting practical impact for on-device reasoning and multi-turn dialogues.

Abstract

As the demand for human-like reasoning, multi-turn dialogues, and long-form responses grows, large language models (LLMs) are increasingly expected to support efficient and effective long-sequence decoding. However, due to limited DRAM capacity, long-seuqence LLM decoding on smartphones is constrained by the key-value cache (KVCache), whose memory footprint increases linearly with sequence length. Retrieval-based methods mitigate DRAM pressure by offloading KVCache to flash and retrieving query-relevant entries through cluster-based indexing. Unfortunately, as decoding progresses, KVCache distribution shifts render static or local cluster updates progressively misaligned, excluding essential entries or fetching redundant ones. These issues are further exacerbated by smartphone-specific limitations in bandwidth, IOPS, and memory capacity. We propose DynaKV, the first adaptive KVCache management approach that jointly addresses accuracy and efficiency for long-sequence decoding on smartphones. DynaKV integrates three key techniques: (1) Migration-Free Cluster Adaptation, which adaptively splits clusters during retrieval without incurring additional transfers; (2) Continuity-Centric Flash Management, which co-locates correlated entries and clusters and employs a dual-head layout for efficient updates; and (3) Memory-Efficient Cache Design, which virtualizes cache space across DRAM and flash and extends replacement policies to align with cluster-level access patterns. Evaluations demonstrate that DynaKV improves retrieval accuracy and reduces end-to-end latency compared to state-of-the-art solutions, achieving average gains of $1.38\times$ in accuracy and $1.47\times$ speedups. Furthermore, the insights of DynaKV naturally extend to other long-context workloads and multi-tier memory hierarchies, underscoring its broader applicability.

Figures (18)

• Figure 1: Illustration of three KVCache cluster management strategies. (a) Static Update: new entries are greedily appended to their nearest existing clusters. (b) Local Update: new entries are re-clustered independently of existing clusters. (c) DynaKV: adaptively appends new entries or splits existing clusters to maintain accuracy and efficiency.
• Figure 2: Two-phase KVCache retrieval process. Offloading KVCache from memory to storage mitigates memory bottlenecks, but shifts the critical bottleneck to I/O operations.
• Figure 3: (a) Two-tier memory hierarchy on smartphones (DRAM and flash), highlighting three key bottlenecks. (b) UFS bandwidth under various continuous I/O sizes, where the limiting factor shifts from IOPS to bandwidth capacity.
• Figure 4: Visualization of KVCache distribution shift during decoding using PCA pca, with Qwen2.5-3B evaluated on the MMLU dataset mmlu. Different colors denote distinct clusters.
• Figure 5: Overview of DynaKV.