Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving

TL;DR

This work addresses the inefficiency of serving online and offline LLM tasks in isolation by introducing Echo, a co-scheduling system that jointly optimizes batch construction, KV cache usage, and execution time estimation. Echo comprises a KV-cache-aware scheduler, a task-aware KV cache manager, and estimation toolkits that predict batch time, memory usage, and system throughput to guide decisions. The key contributions include a plan-generation/selection approach that leverages last-iteration batch information, a reference-count based KV cache eviction policy with thresholds to protect bursty online requests, and a simulation-based estimator suite that enables resource provisioning. Evaluation on real-world traces shows up to 3.3x improvements in offline throughput while satisfying online SLOs, driven by higher KV-cache hit rates (e.g., 78.6% on certain offline workloads) and smarter batch regularity. The findings demonstrate that co-scheduling online/offline tasks with cache-awareness can meaningfully improve resource utilization and cost-effectiveness for large-scale LLM serving.

Abstract

Large language models have been widely deployed in various applications, encompassing both interactive online tasks and batched offline tasks. Given the burstiness and latency sensitivity of online tasks, over-provisioning resources is common practice. This allows for the integration of latency-insensitive offline tasks during periods of low online load, enhancing resource utilization. However, strategically serving online and offline tasks through a preemption mechanism fails to fully leverage the flexibility of offline tasks and suffers from KV cache recomputation and irregular workloads. In this paper, we introduce Echo, a collaborative online-offline task serving system, including a scheduler, a KV cache manager, and estimation toolkits. The scheduler and KV cache manager work tightly to maximize the throughput of offline tasks, while the estimator further predicts execution time to ensure online task SLOs. The scheduler leverages the batch information of last iteration to reduce the search space for finding the optimal schedule. The KV cache manager sets the priority of the KV cache based on the type of tasks and the opportunity of prefix sharing to reduce the recomputation. Finally, the estimation toolkits predict the execution time, future memory consumption, and the throughput of offline tasks to guide the scheduler, KV cache manager, and the system deployer. Evaluation based on real-world workloads demonstrates that Echo can increase offline task throughput by up to $3.3\times$, while satisfying online task SLOs.

Figures (11)

• Figure 1: Batching strategy.
• Figure 2: A 24-hour trace of a typical online task.
• Figure 3: Overview of Echo.
• Figure 4: Different strategies.
• Figure 5: Impact of threshold on KV cache management.