PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices

TL;DR

PerCache addresses the high latency of mobile RAG by introducing a predictive hierarchical cache that jointly stores QA banks and QKV caches, enabling cross-layer reuse of intermediate results. It uses on-device query prediction from knowledge abstracts and history to pre-populate caches during idle time and employs a cache scheduler with cross-layer cache conversion to adapt to dynamic resource conditions. The approach yields substantial latency reductions (up to $34.4\%$) and improved cache hit rates across multiple datasets and mobile devices, while maintaining output quality. This work demonstrates practical, privacy-preserving acceleration for personalized mobile RAG with broad applicability to on-device LLM inference.

Abstract

Retrieval-augmented generation (RAG) has been extensively used as a de facto paradigm in various large language model (LLM)-driven applications on mobile devices, such as mobile assistants leveraging personal emails or meeting records. However, due to the lengthy prompts and the resource constraints, mobile RAG systems exhibit significantly high response latency. On this issue, one promising approach is to reuse intermediate computational results across different queries to eliminate redundant computation. But most existing approaches, such as KV cache reuse and semantic cache reuse, are designed for cloud settings and perform poorly, overlooking the distinctive characteristics of mobile RAG. We propose PerCache, a novel hierarchical cache solution designed for reducing end-to-end latency of personalized RAG applications on mobile platforms. PerCache adopts a hierarchical architecture that progressively matches similar queries and QKV cache to maximize the reuse of intermediate results at different computing stages. To improve cache hit rate, PerCache applies a predictive method to populate cache with queries that are likely to be raised in the future. In addition, PerCache can adapt its configurations to dynamic system loads, aiming at maximizing the caching utility with minimal resource consumption. We implement PerCache on top of an existing mobile LLM inference engine with commodity mobile phones. Extensive evaluations show that PerCache can surpass the best-performing baseline by 34.4% latency reduction across various applications and maintain optimal latency performance under dynamic resource changes.

Figures (28)

• Figure 1: Workflow of mobile RAG applications.
• Figure 2: Pairwise semantic similarity of queries from two distinct users in the Email and Dialog datasets.
• Figure 3: Probability density distribution of knowledge chunk retrieval frequencies.
• Figure 4: Inference latency breakdown of Llama-3.2-3B on Google Pixel 7 and NVIDIA RTX A6000.
• Figure 5: Prefix overlap degree of retrieved chunks.