BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure

TL;DR

BanaServe tackles the inefficiencies of disaggregated LLM serving by decoupling resource allocation from state placement through a Global KV Cache Store and two granular migration mechanisms (layer-level and attention-level). Its adaptive migration and load-aware scheduling continuously rebalance compute and memory across prefill and decode, hiding communication latency with a three-stage KV cache pipeline and enabling purely load-driven routing. The framework achieves substantial throughput gains (up to 3.9× over vLLM and 2.8× over DistServe) and latency reductions (up to 78.4%), validating its effectiveness under dynamic workloads and across short- and long-context scenarios. These results indicate BanaServe’s practical potential for production AI infrastructure, especially in heterogeneous, bursty environments, while future work could address hardware heterogeneity, predictive orchestration, WAN-scale KV cache sharing, and production-grade deployment tooling.

Abstract

Large language models (LLMs) are increasingly deployed in AI infrastructure, driving the need for high throughput, resource efficient serving systems. Disaggregated LLM serving, which separates prompt prefill from auto-regressive decode, has emerged as a promising architecture by isolating their heterogeneous compute and memory demands. However, current disaggregated systems face three key limitations: (i) static resource allocation cannot adapt to highly dynamic workloads, causing over-provisioning that wastes resources or under-provisioning that violates service level objectives (SLOs); (ii) inherent load imbalance between prefill and decode stages, where prefill is compute-bound and decode is memory-bound, causes under-utilization in one tier while the other becomes a bottleneck; and (iii) prefix cache aware routing skews load distribution, as high cache hit rate prefill nodes attract disproportionately more requests, further degrading balance and efficiency. To address these issues, we present BanaServe, a dynamic orchestration framework that continuously rebalances computational and memory resources across prefill and decode instances while eliminating hotspots induced by cache. BanaServe introduces layer level weight migration, attention level Key Value Cache (KV Cache) migration, and Global KV Cache Store sharing with layer wise overlapped transmission, enabling both coarse grained (layer level) and fine grained (attention level) load redistribution with minimal latency overhead. These mechanisms allow routers to perform purely load aware scheduling, unconstrained by cache placement. Compared to vLLM, BanaServe achieves 1.2x-3.9x higher throughput with 3.9%-78.4% lower total processing time, and outperforms DistServe by 1.1x-2.8x in throughput with 1.4%-70.1% latency reduction.

Figures (10)

• Figure 2: Two empirically validated limitations in current LLM serving architectures. (a) Prefix-cache-aware routing leads to persistent load skew, redundant cache storage and repeated computation, as high-hit-rate instances attract more requests while others remain underutilized. (b) PD disaggregation exhibits an inherent imbalance across compute, memory and bandwidth resources, resulting in under-utilization of one dimension while another becomes a bottleneck.
• Figure 3: Layer level migration in BanaServe: A set of adjacent Transformer layers is dynamically reallocated across prefill and decode GPUs, with both layer weights $W_{\ell}$ and KV Cache $\mathcal{K}\mathcal{V}_{\ell}$ migrated. Local execution resumes after payload transfer, enabling coarse grained compute and memory rebalancing while preserving output correctness. Device #0 and Device #1 process different segments in parallel until migration completes, avoiding service disruption.
• Figure 4: Attention level migration in BanaServe. KV Cache is split along the attention head dimension into $K^{(1)},V^{(1)}$ (hot GPU) and $K^{(2)},V^{(2)}$ (cold GPU). Local attention (Atten for Local Seq) and offloaded attention (Atten for Offload Seq, preceded by FC1 and followed by FC2) are computed in parallel. Only the partial softmax denominator $\ell^{(1)}$ and output $O^{(1)}$ are exchanged across devices, enabling minimal data transfer and mitigating hotspots with negligible service disruption.
• Figure 5: KV Cache management and reuse in BanaServe. Prefill instances perform incremental prefill on incoming requests, reusing cached prefixes when available. Prefix KV Cache and newly generated incremental KV Cache are stored in a shared CPU/SSD-backed KV store, enabling both cache sharing across prefill GPUs and complete KV Cache assembly. During decoding, decode instances retrieve the full KV Cache from the shared store and reuse it for token generation, avoiding redundant computation and supporting efficient prefill and decode execution.
• Figure 6: Validation of the three-stage layer-wise KV Cache pipeline in BanaServe. The top section lists hardware parameters and measured latencies for a representative workload. Under a 50% average prefix cache hit rate, the per-layer forward time ($4.22$ ms) exceeds the per-layer KV Cache transfer time ($0.082$ ms), ensuring effective overlap. The timeline shows the execution for three layers ($L1$–$L3$): while the GPU executes forward computation for $L_i$, the host-to-device (HtoD) channel fetches KV Cache for $L_{i+1}$ and the device-to-host (DtoH) channel stores cache from $L_{i-1}$. This concurrency hides communication latency and enables transparent prefill execution.