MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference

TL;DR

MPCache addresses the latency and communication bottlenecks of private LLM inference under MPC by designing an MPC-friendly KV cache eviction framework that combines a one-shot look-once static eviction with a query-aware dynamic selection. The approach leverages MPC-friendly similarity approximation, hierarchical KV cache clustering, and cross-layer index-sharing to drastically reduce attention-related costs without retraining. Empirical results on LongBench and XSUM show 1.59–2.01× latency reductions and up to 8.37× communication savings across sequence lengths, outperforming prior KV eviction baselines. The framework enables scalable, secure private inference with minimal performance sacrifice, advancing practical deployment of MPC-based LLM services.

Abstract

Private large language model (LLM) inference based on secure multi-party computation (MPC) achieves formal data privacy protection but suffers from significant latency overhead, especially for long input sequences. While key-value (KV) cache eviction and sparse attention algorithms have been proposed for efficient LLM inference in plaintext, they are not designed for MPC and cannot benefit private LLM inference directly. In this paper, we propose an accurate and MPC-friendly KV cache eviction framework, dubbed MPCache, building on the observation that historical tokens in a long sequence may have different effects on the downstream decoding. Hence, MPCache combines a look-once static eviction algorithm to discard unimportant KV cache and a query-aware dynamic selection algorithm to activate only a small subset of KV cache for attention computation. MPCache further incorporates a series of optimizations for efficient dynamic KV cache selection, including MPC-friendly similarity approximation, hierarchical KV cache clustering, and cross-layer index-sharing strategy. Extensive experiments demonstrate that MPCache consistently outperforms prior-art KV cache eviction baselines across different generation tasks and achieves 1.8 ~ 2.01x and 3.39 ~ 8.37x decoding latency and communication reduction on different sequence lengths, respectively.

Figures (21)

• Figure 1: (a) MPC-based LLM inference. (b) Breakdown of decoding latency and communication with a sequence length of 512. Attention dominates the overhead for both 3PC and 2PC protocols. (c) The cost of Softmax scales with the sequence length rapidly. (d) Inference cost before and after KV cache eviction. Blocks in slash indicate the extra overhead introduced by eviction.
• Figure 2: (Upper) token types in attention maps where ✓ means the token is selected and ✗ means the token is not selected. (Lower) three types can be observed in attention map with more tokens.
• Figure 3: Motivating inspirations. (a) Statically evicting 60% tokens during the prefill stage still maintains the performance; (b) less than 20% tokens contribute to decoding; (c) cross-layer commonality among different numbers of adjacent layers.
• Figure 4: Overview of our proposed MPCache.
• Figure 5: The illustration of static eviction.