CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control

TL;DR

This work addresses the bottleneck of middle-phase thrashing in agentic batch inference, where long-horizon agent contexts exhaust GPU KV caches and degrade throughput. It introduces CONCUR, a lightweight agent-level admission control layer that uses a congestion-control–inspired AIMD loop and runtime signals (KV-cache usage $U_t$ and hit rate $H_t$) to regulate the number of concurrently active agents via the update $W_{t+1} = W_t + \alpha$ if $U_t < U_{low}W_t$ and $W_{t+1} = \beta W_t$ if $U_t > U_{high} \land H_t < H_{thresh}W_t$, thereby bounding cache pressure while preserving execution continuity. The approach preserves agent continuity, reduces recomputation, and improves throughput across large models (e.g., up to $4.09\times$ on Qwen3-32B and $1.9\times$ on DeepSeek-V3) compared to baselines. The work demonstrates that proactive, agent-level flow control can stabilize memory usage and unlock scalable, high-throughput autonomous agent inference in real-world serving stacks. The method's principled design and empirical results suggest broad applicability to diverse agent-based inference workloads and LLM serving systems.

Abstract

Batch inference for agentic workloads stresses the GPU key-value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory capacity is exhausted. We identify this phenomenon as middle-phase thrashing, a previously under-characterized pathology in which cache efficiency collapses as long-lived agents accumulate state over time. We argue that mitigating this pathology requires moving beyond reactive, request-level cache management to proactive, agent-level admission control. Drawing inspiration from congestion control in distributed systems, we view the KV cache as a shared resource whose efficient utilization depends on feedback-driven regulation. Based on this insight, we present CONCUR, a lightweight control layer that regulates agent admission to bound aggregate cache pressure while preserving execution continuity. CONCUR adapts a cache-aware control algorithm to dynamically adjust the number of active agents using runtime cache signals. Across large models and real-world agent workloads, CONCUR prevents middle-phase thrashing and improves batch inference throughput by up to 4.09x on Qwen3-32B and 1.9x on DeepSeek-V3, while remaining compatible with existing LLM serving systems.

Figures (6)

• Figure 1: (a) (b) Input length and KV cache memory consumption growth across 10 generation steps for DeepSeek V3 and Qwen3-32B. (c) Comparison of GPU-to-CPU KV cache offload latency versus prefill-based recomputation latency for DeepSeek-V3 (6.67 GB cache per request, 4096 tokens) under varying concurrency levels.
• Figure 2: Three-agent workflow illustrating middle-phase thrashing and agent-level admission control. (a) LRU eviction causes paused agents to lose KV cache entries, triggering repeated recomputation and middle-phase thrashing. (b) Agent-level admission control bounds concurrency, preventing cache overcommitment and eviction-induced recomputation.
• Figure 3: Middle-phase thrashing in real-world agentic batch inference. (a) End-to-end KV cache usage over time when running a benchmark on DeepSeek-V3. The trace exhibits a characteristic three-phase execution pattern in batch agent inference. (b) End-to-end latency breakdown of the same run, showing the fraction of time spent in prefill and decode, and additional recomputation overhead induced by KV cache thrashing during the middle phase.
• Figure 4: System Overview.
• Figure 5: Temporal dynamics of KV cache during large-batch offline agentic inference for Qwen3-32B under constraint resources (Batch 256, TP=2, 2 GPUs).