Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications

Zhuohang Bian, Feiyang Wu, Teng Ma, Youwei Zhuo

TL;DR

Tokencake addresses the dual challenges of time underutilization and space contention in KV Cache for LLM-based multi-agent workflows. It introduces an agent-aware framework with a Frontend API, a Space Scheduler for dynamic memory partitioning, and a Time Scheduler for event-driven offload and predictive upload, enabling proactive KV Cache management during long tool calls. The paper demonstrates substantial latency reductions (up to 47.06%) and memory utilization improvements (up to 16.9%) over vLLM under realistic multi-agent workloads, validating the benefits of co-optimizing scheduling with memory management. This work significantly improves the practicality and efficiency of deploying complex, tool-using LLM agents in production settings, by efficiently utilizing GPU memory while preserving responsiveness.”

Abstract

Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that use external function calls. This workload creates severe performance challenges for the KV Cache: space contention leads to the eviction of critical agents' caches and time underutilization leaves the cache of agents stalled on long-running tool calls idling in GPU memory. We present Tokencake, a KV-Cache-centric serving framework that co-optimizes scheduling and memory management with an agent-aware design. Tokencake's Space Scheduler uses dynamic memory partitioning to shield critical agents from contention, while its Time Scheduler employs a proactive offload and predictive upload mechanism to repurpose GPU memory during function call stalls. Our evaluation on representative multi-agent benchmarks shows that Tokencake can reduce end-to-end latency by over 47.06%, improve effective GPU memory utilization by up to 16.9% compared to vLLM.

Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications

TL;DR

Tokencake addresses the dual challenges of time underutilization and space contention in KV Cache for LLM-based multi-agent workflows. It introduces an agent-aware framework with a Frontend API, a Space Scheduler for dynamic memory partitioning, and a Time Scheduler for event-driven offload and predictive upload, enabling proactive KV Cache management during long tool calls. The paper demonstrates substantial latency reductions (up to 47.06%) and memory utilization improvements (up to 16.9%) over vLLM under realistic multi-agent workloads, validating the benefits of co-optimizing scheduling with memory management. This work significantly improves the practicality and efficiency of deploying complex, tool-using LLM agents in production settings, by efficiently utilizing GPU memory while preserving responsiveness.”

Abstract

Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that use external function calls. This workload creates severe performance challenges for the KV Cache: space contention leads to the eviction of critical agents' caches and time underutilization leaves the cache of agents stalled on long-running tool calls idling in GPU memory. We present Tokencake, a KV-Cache-centric serving framework that co-optimizes scheduling and memory management with an agent-aware design. Tokencake's Space Scheduler uses dynamic memory partitioning to shield critical agents from contention, while its Time Scheduler employs a proactive offload and predictive upload mechanism to repurpose GPU memory during function call stalls. Our evaluation on representative multi-agent benchmarks shows that Tokencake can reduce end-to-end latency by over 47.06%, improve effective GPU memory utilization by up to 16.9% compared to vLLM.
Paper Structure (24 sections, 4 equations, 14 figures, 3 tables, 2 algorithms)

This paper contains 24 sections, 4 equations, 14 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. LLM-based Multi-Agent Applications
  4. Challenges
  5. Limitations of Existing Serving Systems
  6. Overview
  7. Frontend API
  8. Coordination between Space and Time Schedulers
  9. The Time Scheduler
  10. Event-Driven Offload and Predictive Upload
  11. Opportunistic Policy for Proactive Offloading
  12. Mitigating Offload and Predictive Upload Overhead
  13. The Space Scheduler
  14. Runtime Control with Dynamic Memory Partitioning
  15. Hybrid Priority Metric for Agent-Awareness
  16. ...and 9 more sections

Figures (14)

  • Figure 1: Example LLM-based Multi-Agent Applications. Each colored box represents a specialized agent. Purple boxes denote function calls to external tools.
  • Figure 2: The Space Contention Problem.
  • Figure 3: The Time Underutilization Problem.
  • Figure 4: Tokencake Overview.
  • Figure 5: Example of defining a multi-agent RAG application with Tokencake's API.
  • ...and 9 more figures