When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judges

TL;DR

The paper investigates judge-centric effects of KV cache reuse in multi-agent LLM systems, showing that reuse can change which candidate the judge selects even when the final outcome is correct, as measured by end-task accuracy ($Acc$). It evaluates several judge-side KV reuse strategies (Naïve, KVCOMM, PAL-KV) across GSM8K, MMLU, and HumanEval using dense prefill as a reference and introduces Judge Consistency Rate ($JCR$) to quantify decision invariance. Attention diagnostics reveal that reuse weakens cross-candidate attention, especially for later candidate blocks, and a masking ablation confirms that cross-candidate interaction is essential for preserving dense-prefill decisions. The findings motivate interaction-preserving reuse and risk-aware gating as future directions to maintain judge reliability and auditability in multi-agent pipelines.

Abstract

Multi-agent LLM systems routinely generate multiple candidate responses that are aggregated by an LLM judge. To reduce the dominant prefill cost in such pipelines, recent work advocates KV cache reuse across partially shared contexts and reports substantial speedups for generation agents. In this work, we show that these efficiency gains do not transfer uniformly to judge-centric inference. Across GSM8K, MMLU, and HumanEval, we find that reuse strategies that are effective for execution agents can severely perturb judge behavior: end-task accuracy may appear stable, yet the judge's selection becomes highly inconsistent with dense prefill. We quantify this risk using Judge Consistency Rate (JCR) and provide diagnostics showing that reuse systematically weakens cross-candidate attention, especially for later candidate blocks. Our ablation further demonstrates that explicit cross-candidate interaction is crucial for preserving dense-prefill decisions. Overall, our results identify a previously overlooked failure mode of KV cache reuse and highlight judge-centric inference as a distinct regime that demands dedicated, risk-aware system design.

Figures (16)

• Figure 1: Illustration of decision non-invariance under judge-side KV cache reuse. Top: dense prefill recomputes the judge KV cache and selects Agent 1. Bottom: KV reuse stitches/modifies cached KV blocks, keeping the final answer correct but changing the selected best agent, despite identical candidate texts.
• Figure 2: Judge-side KV cache construction for multi-candidate judging. Dense prefill recomputes the full judge cache, while Naïve Reuse aligns and stitches execution-side candidate KV chunks. KVCOMM retrieves anchor-based cache offsets to correct reused chunks, and PAL-KV pools anchors across agents for offset retrieval.
• Figure 3: Relative attention mass over regions (prefix and candidate slots) under different KV reuse methods.
• Figure 4: Ablations under shuffle: judge-side decision non-invariance persists. Varying anchor pool size, candidate count, or model size does not reliably restore JCR for KV reuse methods.
• Figure 5: Jaccard similarity of selected Top-$k$% tokens between the small and large models.