Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks

TL;DR

The paper evaluates prompt caching for long-horizon LLM agents across three major providers (OpenAI, Anthropic, Google) and four flagship models, using the DeepResearchBench benchmark to assess cost and latency. It compares three caching strategies—full context, system prompt only, and exclude tool results—against a no-cache baseline, employing UUID-based boundary control to isolate prefix reuse. Results show substantial cost savings (roughly 28–81% depending on model and strategy) and TTFT improvements (approximately 6–31%), with system-prompt-focused caching delivering the most consistent benefits. The authors argue that strategic cache boundary control, rather than naive full-context caching, yields reliable performance gains, and they offer practical guidance for deploying prompt caching in production agentic systems, including considerations for tool calls and provider-specific variability.

Abstract

Recent advancements in Large Language Model (LLM) agents have enabled complex multi-turn agentic tasks requiring extensive tool calling, where conversations can span dozens of API calls with increasingly large context windows. However, although major LLM providers offer prompt caching to reduce cost and latency, its benefits for agentic workloads remain underexplored in the research literature. To our knowledge, no prior work quantifies these cost savings or compares caching strategies for multi-turn agentic tasks. We present a comprehensive evaluation of prompt caching across three major LLM providers (OpenAI, Anthropic, and Google) and compare three caching strategies, including full context caching, system prompt only caching, and caching that excludes dynamic tool results. We evaluate on DeepResearchBench, a multi-turn agentic benchmark where agents autonomously execute real-world web search tool calls to answer complex research questions, measuring both API cost and time to first token (TTFT) across over 500 agent sessions with 10,000-token system prompts. Our results demonstrate that prompt caching reduces API costs by 45-80% and improves time to first token by 13-31% across providers. We find that strategic prompt cache block control, such as placing dynamic content at the end of the system prompt, avoiding dynamic traditional function calling, and excluding dynamic tool results, provides more consistent benefits than naive full-context caching, which can paradoxically increase latency. Our analysis reveals nuanced variations in caching behavior across providers, and we provide practical guidance for implementing prompt caching in production agentic systems.

Figures (8)

• Figure 1: Prompt caching benefits (best cache mode per model). Percentage reduction in API cost and time to first token (TTFT) relative to a no-cache baseline. For each model, results correspond to the best-performing cache strategy. Asterisks denote statistically significant TTFT improvements ($p < 0.05$).
• Figure 2: Prompt caching impact for normalized cost and time to first token (TTFT). Results use the system prompt only caching strategy. The no-cache baseline is normalized to 100% and lower values indicate better performance.
• Figure 3: Normalized cost and time to first token (TTFT) distributions by model and cache strategy. The no-cache baseline is normalized to 100% and lower values indicate better performance. Cost reductions are consistent across cache strategies, while TTFT improvements vary significantly, with full-context caching sometimes underperforming more selective strategies.
• Figure 4: Prompt caching requires exact prefix matches. Different shades represent message types in agentic conversations: brightest (system prompt), light gray (human messages), medium gray (AI messages), darker gray (tool calls), and darkest (tool results). Cache hit: The prompt prefix matches a previously seen request exactly, so cached KV tensors are reused (green) and only new tokens appended at the end require computation (gray). Cache miss: Any difference in the prefix---even a single token at the beginning (orange)---prevents cache reuse, forcing full recomputation of all tokens (gray).
• Figure 5: No Cache (Baseline): A unique UUID prepended to the start of the system prompt ensures no prefix match is possible with any prior request, forcing full recomputation of all tokens every time.