Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

TL;DR

This work tackles inefficiencies in KV-cache eviction within multi-turn LLM agent workloads, where tool calls and interleaved reasoning degrade throughput. It introduces Continuum, a TTL-based KV-cache retention mechanism that pins GPU KV-cache during tool calls, guided by a cost–benefit utility model and TTL-aware scheduling that preserves program-level FCFS. By estimating reload costs, per-turn queueing reductions, and tool-call distributions, Continuum computes an optimal TTL $\tau^*$ to balance memory usage and latency savings, even under cold-start and variable tool latencies. Evaluations on SWE-Bench, BFCL, and internal testbeds show consistent reductions in delay ($1.12\times$–$3.66\times$) and throughput improvements ($1.10\times$–$3.22\times$), with up to $8.18\times$ gains in real workloads, demonstrating the practical impact of principled, tool-aware KV-cache management for complex agent workflows.

Abstract

KV cache management is essential for efficient LLM inference. To maximize utilization, existing inference engines evict finished requests' KV cache if new requests are waiting. This policy breaks for agentic workloads, which interleave LLM calls with tools, introducing pauses that prevent effective KV reuse across turns. Since some tool calls have much shorter durations than human response multi-turn chatbot, it would be promising to retain the KV cache in during these tools. However, there are many challenges. First, we need to consider both the potential cost of recomputation or reloading (if CPU offloading enabled) and the increasing queueing delays after eviction from GPU. Second, due to the internal variance of tool call durations, we need the method to remain robust under limited predictability of tool call durations. We present Continuum, a serving system to optimize job completion time for multi-turn agent workloads by introducing time-to-live mechanism for KV cache retaining. For LLM request that generates a tool call, Continuum selectively pins the KV cache in GPU memory with a time-to-live value determined by considering both the reload cost and ordering preserve benefit of retaining KV cache. Moreover, when the TTL expires, the KV cache can be automatically evicted to free up GPU memory, providing robust performance under edge cases. When combined with program-level first-come-first-serve, Continuum preserves multi-turn continuity, and reduces delay for complex agentic workflows. Our evaluation on real-world agentic workloads (SWE-Bench and BFCL) with Llama-3.1 8B/70B shows that Continuum significantly improves the average job completion times and its improvement scales with turn number increase. We release a preview version at: https://github.com/Hanchenli/vllm-continuum

Figures (15)

• Figure 1: Two main failures of previous work on handling tool calls, where red blocks represent unnecessary overhead caused by suboptimal scheduling and KV cache management: Even with CPU offloading, agent programs still suffer from queueing delay after KV cache eviction.
• Figure 2: Illustrative example of a SWE-Agent. The agent resolves a software engineering bug step by step with tool calls in the middle. These tool calls have different durations and breaks the continuity of the LLM inference.
• Figure 3: Workload characteristics of agentic workloads SWE-Bench and BFCL as used in Sec \ref{['sec:evaluation']}. As the number of steps increase, the requests are closer to finish.
• Figure 4: vLLM and InferCept incurs high waiting time for each agentic program even under CPU offloading due to per-turn queueing delay.
• Figure 5: Functions' execution time can be extremely long-tailed. Slowest 10% of fetch_url account for 52.5% of the total delay, while slowest 10% of cd account for 94.1%.