EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration

TL;DR

EcoServe addresses the cost and efficiency gap in large-scale LLM serving on commodity networks by introducing a Partially Disaggregated ($PaDG$) strategy that temporally disaggregates prefill and decode within each instance and coordinates requests across a macro-instance via rolling activation. The system combines an adaptive intra- and inter-instance scheduling framework with a mitosis-based scaling approach, enabling fine-grained capacity adjustments while maintaining SLOs. Empirical results on 30B–70B scale models show substantial goodput gains over NoDG and FuDG baselines, with notable improvements in throughput under tight SLOs and across diverse datasets and hardware configurations. The work demonstrates that careful orchestration and scalable, low-overhead migration can achieve high efficiency and simplicity on commodity hardware, making cost-effective LLM serving more practical at scale.

Abstract

Existing LLM serving strategies can be categorized based on whether prefill and decode phases are disaggregated: non-disaggregated (NoDG) or fully disaggregated (FuDG). However, the NoDG strategy leads to strong prefill-decode interference and the FuDG strategy highly relies on high-performance interconnects, making them less cost-effective. We introduce EcoServe, a system that enables cost-effective LLM serving on clusters with commodity interconnects. EcoServe is built on the partially disaggregated (PaDG) strategy, applying temporal disaggregation and rolling activation for proactive intra- and inter-instance scheduling. It first disaggregates the prefill and decode phases along the time dimension within a single instance to mitigate inter-phase interference and enhance throughput. Next, it coordinates multiple instances and cyclically activates them to ensure the continuous availability of prefill processing, thereby improving latency. Thus, EcoServe's basic serving unit is the macro instance, within which multiple instances collaborate. It further integrates an adaptive scheduling algorithm to route requests in a macro instance and a mitosis scaling approach to enable fine-grained capacity scaling. Beyond delivering high goodput, EcoServe excels in load balancing, hardware cost, parallelism compatibility, and even engineering simplicity compared to existing solutions. When serving 30B- and 70B-scale models on a production-level cluster with 32 NVIDIA L20 GPUs using commodity Ethernet, EcoServe averagely improves goodput by 82.49%, 86.17%, 122.76%, and 126.96% over four representative NoDG and FuDG systems.

Figures (11)

• Figure 1: The NoDG, FuDG, and PaDG strategies.
• Figure 2: LLM autoregressive decoding process.
• Figure 3: Tensor parallelism and pipeline parallelism.
• Figure 4: Pipeline bubbles.
• Figure 5: EcoServe Architecture Overview.