SYMPHONY: Improving Memory Management for LLM Inference Workloads

TL;DR

Multi-turn LLM serving introduces heavy KV-cache memory requirements that are poorly served by recomputation or simple offloading, leading to wasted compute or cluster-level load imbalances. SYMPHONY proposes advisory requests as lightweight hints about imminent requests and couples a Scheduler with a Node Manager to migrate KV caches across hierarchical memory while scheduling at the per-request level. Empirical results on LLaMa-3.1 8B and LLaMa-2-13B using ShareGPT traces show substantial gains: up to 2.4x shorter end-to-end latency over vLLM and up to 4x more requests served with only about 1.3% latency inflation. The approach is compatible with existing schedulers, enables flexible policies such as prioritization and tiered storage, and demonstrates the practical viability of predictive cache management for scalable, low-latency multi-turn LLM inference.

Abstract

Large Language Models (LLMs) are increasingly being deployed in applications such as chatbots, code editors, and conversational agents. A key feature of LLMs is their ability to engage in multi-turn interactions with humans or external tools, enabling a wide range of tasks. Each new request in a multi-turn interaction depends on the intermediate state, specifically the key-value (K,V) caches, from previous requests in the ongoing interaction. Existing serving engines either recompute the K,V caches or offload them to main memory. Profiling reveals that recomputation can result in over 99% of processed tokens being redundant. On the other hand, offloading K,V caches from GPU memory makes inference serving stateful, leading to load imbalances across the cluster. To address these challenges, we developed SYMPHONY. SYMPHONY leverages the observation that multi-turn work loads provide additional hints that allow K,V caches to be migrated off the critical serving path. By utilizing these hints, SYMPHONY dynamically migrates K,V caches to enable finegrained scheduling of inference requests. Our experiments demonstrate that SYMPHONY can handle over 8x the number of requests compared to state-of-the-art baselines, with a similar latency profile.

Figures (18)

• Figure 1: Load imbalance in stateful LLM Serving: We serve 1024 concurrent users with 8 GPUs with Infercept a popular system that offloads K,V caches to host memory. We observe that the difference between the most loaded GPU and the least GPU can be 300 requests. This indicates that stateful LLM serving can lead to high load imbalance.
• Figure 2: Scaling Behavior: We plot Average Time Per Output Token(TPOT) when serving LLaMa-3 8B on A100. We observe that the time taken to produce one token drastically increases as the number of concurrent requests increases on the GPU. Processing 8 concurrent requests vs 32 concurrent requests can lead to lead to almost double the latency.
• Figure 3: Chatbot Interaction: For each chatbot response it needs access to K,V cache values for all previous interactions, i.e., to get Response 2, we need K,V cache access to all previous queries and responses.
• Figure 4: CDF of the number of turns in a chatbot: Distribution of the number of turns in the ShareGPT dataset. We observe that more than 73% of conversations are multi-turn conversations.
• Figure 5: Comparing recompute and swap: The above figure shows the difference between recompute and swap approaches. Using swapping can reduce the prefill time.