RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

Abstract

The increasing complexity of AI tasks has shifted the paradigm from monolithic models toward multi-agent large language model (LLM) systems. However, these collaborative architectures introduce a critical bottleneck: redundant prefill computation for shared content generated by previous agents, which significantly increases KV cache memory usage and time-to-first-token (TTFT). While various KV cache methods have been proposed to mitigate prefill redundancy, they either fail to maintain accuracy on agent-generated outputs or exhibit low reuse rates due to rigid constraints. We present RelayCaching, a training-free inference method that directly reuses decoding phase KV caches from previous agents in subsequent prefill phases. Our key insight is that KV caches for identical content are highly consistent across phases, while prefix-induced deviations are sparse and localized within a limited range of layers and token positions. By selectively recomputing KV caches at these positions, RelayCaching preserves model accuracy with minimal overhead, yielding a superior accuracy-efficiency trade-off over existing methods. Experiments on diverse collaborative LLM tasks spanning mathematical reasoning, general knowledge, and code generation demonstrate that RelayCaching achieves over 80% KV cache reuse, reduces TTFT by up to $4.7\times$ compared to the standard pipeline, all with negligible accuracy degradation.

Figures (10)

• Figure 1: Overview of KV cache reuse in multi-agent LLM collaboration. Top: A typical workflow where an Architect generates {spec}, a Developer produces {code} based on it, and a Reviewer evaluates the result, so each agent's output becomes the next agent's input, causing cascading redundant prefill. Bottom: Comparison of caching strategies. (a) Prefix Caching fails due to prefix mismatch; (b) Pre-computed Caching cannot handle dynamic content; (c) RelayCaching enables decode-to-prefill KV reuse via selective rectification.
• Figure 2: Macro-level KV similarity. Average cosine and norm similarity between Decoding KV caches and Full-Prefill KV caches across layers and tokens on 2WikiMQA, compared against Random KV and Independent KV.
• Figure 3: Layer-wise structured deviations.(a) Layer-wise similarity between decoding and full-prefill KV caches, highlighting a U-shaped value cosine similarity profile across layers. (b) Relative recovery of value cosine similarity for tokens after the reuse position when decoding KV caches are replaced with their full-prefill counterparts either up to or from a given layer index.
• Figure 4: Token-wise structured deviations.(a) Token-wise similarity between decoding and full-prefill KV caches, averaged over layers, revealing a sparse set of high-deviation positions. (b) Spearman rank correlation of token-wise value cosine deviations between adjacent layers, showing that high-deviation positions, once emerged, tend to persist.
• Figure 5: Overview of RelayCaching. (1) An offline layer-range profiler analyzes calibration data to identify a critical layer range $[L_{\text{start}}, L_{\text{end}}]$ from the similarity profile and a detection layer $L_{\text{det}}$ from the inter-layer correlation profile. (2) During stage-$(i{-}1)$ decoding, the cache manager stores pre-RoPE KV caches, saves hidden states at $L_{\text{start}}$, and accumulates influence scores $s_{\text{inf}}(j)$. (3) At stage-$i$ prefilling, RelayCaching re-applies RoPE to align reused KV caches to new positions, performs token selection at $L_{\text{det}}$ via combined deviation- and influence-based criteria, and finally applies sparse rectification on selected tokens within $[L_{\text{start}}, L_{\text{end}}]$.