KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems

TL;DR

This work tackles the costly prefill stage in online multi-agent LLM systems, where overlapping contexts must be reprocessed for every agent. It introduces KVComm, a training-free framework that enables adaptive KV-cache reuse across diverse prefix contexts by maintaining an online anchor pool and aligning caches via RoPE de-rotation and offset interpolation. The approach is backed by theoretical insights into KV-cache proximity and drift, and validated on RAG, math reasoning, and programming tasks, achieving up to ~7.8x TTFT speedups and reuse rates up to ~95% with minimal accuracy loss. KVComm thus offers a practical pathway to scalable, real-time multi-agent collaboration with open architectures, without retraining or architectural changes.

Abstract

Multi-agent large language model (LLM) systems are increasingly adopted for complex language processing tasks that require communication and coordination among agents. However, these systems often suffer substantial overhead from repeated reprocessing of overlapping contexts across agents. In typical pipelines, once an agent receives a message from its predecessor, the full context-including prior turns-must be reprocessed from scratch, leading to inefficient processing. While key-value (KV) caching is an effective solution for avoiding redundant computation in single-agent settings where prefixes remain unchanged, it cannot be directly reused in multi-agent scenarios due to diverging prefixes introduced by agent-specific context extensions. We identify that the core challenge lies in the offset variance of KV-caches across agents. To address this, we propose KVCOMM, a training-free framework that enables efficient prefilling in multi-agent inference by reusing KV-caches and aligning cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM estimates and adjusts KV-caches for shared content by referencing a pool of cached examples-termed anchors-that store observed cache deviations under varying prefixes. The anchor pool is maintained and updated online, allowing dynamic adaptation to distinct user requests and context structures. KVCOMM achieves over 70% reuse rate across diverse multi-agent workloads, including retrieval-augmented generation, math reasoning, and collaborative coding tasks, all without quality degradation. Particularly, when each fully-connected agent receives 1K input tokens with 512 prefix tokens and 512 output tokens under a five-agent setting, KVCOMM achieves up to 7.8x speedup compared to the standard prefill pipeline, reducing TTFT from ~430 ms to ~55 ms.

Figures (22)

• Figure 1: (a) Mean KV-cache offset (measured by $\ell_2$ norm) from the base-context cache for the same token across ten distinct prefixes. Shaded regions indicate standard deviation across these prefixes. (b) KV-cache offset comparison between two embedding-similar tokens from the base-context caches when both tokens are prefixed with a new context. "$\Delta K1$ w/ rot" and "$\Delta K2$ w/ rot" represent Key offsets of two tokens with position alignment, respectively. "$\Delta K1$ w/o rot" and "$\Delta K2$ w/o rot" refer to Key offsets without alignment. "$\Delta V1$" and "$\Delta V2$" denote Value offsets of two tokens.
• Figure 2: Comparisons with existing KV-cache reuse methods. (Left) The original no-cache-reuse baseline method densely prefills the tokens of all requests. (Middle) Selective recomputation methods yao2025cacheblendliu2024droidspeak select the most critical part of KV-cache for recomputation and reuse the remaining cache of each request. (Right) KVComm reuses all KV-caches of the shared context and introduces context-aware cache offsets to align with different prefix contexts, where the context-aware offset refers to the KV-cache deviation induced by the changed prefix context. Such an offset is approximated by the ground-truth ones of previous similar requests. After approximation, the model runner directly starts decoding without prefill.
• Figure 2: Per‑agent TTFT breakdown and speedup. (Prefix token length per agent: 512; Output token length: 512; Model: Llama-3.1; #Agents = 5)
• Figure 3: Overview of the KVComm framework in a three-agent scenario. Initially, each agent precomputes and stores the KV-cache of prefix segments from its prompt template for future reuse. At runtime, upon receiving a new request, agents check placeholder shareability and query matched anchors. Matched anchors help approximate KV-cache offsets for placeholders and subsequent prefixes through embedding-based interpolation. The matching criteria consider length compatibility and embedding proximity. The updated KV-caches are concatenated for efficient decoding. After decoding, the KV-cache Communication module assesses newly generated caches for potential sharing with other agents based on the established matching rules.
• Figure 4: (a)(b) KV-cache proximity experiments for token pairs sharing identical prefixes: (a) shows the KV-cache distances (measured by $\ell_2$ norm) across layers of the token pairs, which are grouped into "near", "mid", and "far" by embedding distance between the two tokens in the token pairs. (b) shows the Spearman spearman1987proof correlation between embedding distances and KV-cache proximity across layers. (c)(d) KV-cache offset proximity experiments for token pairs prefixed by two distinct contexts: (c) shows the layer-wise KV-cache offset distances between tokens grouped by embedding proximity. (d) shows the Spearman correlation between embedding distance and KV-cache offset proximity. Experimental details are shown in Appendix \ref{['app:stats']}.