DiSCo: Device-Server Collaborative LLM-Based Text Streaming Services

TL;DR

DiSCo tackles the QoE-cost tension in real-time LLM text streaming by introducing a device-server cooperative scheduler that adaptively routes and migrates token generation. It combines a unified cost model with a dispatch controller (device- or server-dominated) and a token-level migration framework with a buffer-based protocol to preserve seamless delivery. Real-world traces from GPT, DeepSeek, and open-source LLaMA-3 benchmarks show substantial TTFT improvements (mean 6-78% and tail 11-52%) and up to 84% cost reductions via migrations, while maintaining stable TBT. This device-server collaboration enables cost-efficient, scalable LLM streaming suitable for diverse mobile and cloud edge deployments, with practical implications for QoE-sensitive applications. $TTFT$, $TBT$, and cost$-$aware routing and migration form the core innovations underpinning DiSCo's performance gains.

Abstract

The rapid rise of large language models (LLMs) in text streaming services has introduced significant cost and Quality of Experience (QoE) challenges in serving millions of daily requests, especially in meeting Time-To-First-Token (TTFT) and Time-Between-Token (TBT) requirements for real-time interactions. Our real-world measurements show that both server-based and on-device deployments struggle to meet diverse QoE demands: server deployments face high costs and last-hop issues (e.g., Internet latency and dynamics), while on-device LLM inference is constrained by resources. We introduce DiSCo, a device-server cooperative scheduler designed to optimize users' QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. DiSCo employs cost-aware scheduling, leveraging the predictable speed of on-device LLM inference with the flexible capacity of server-based inference to dispatch requests on the fly, while introducing a token-level migration mechanism to ensure consistent token delivery during migration. Evaluations on real-world workloads -- including commercial services like OpenAI GPT and DeepSeek, and open-source deployments such as LLaMA3 -- show that DiSCo can improve users' QoE by reducing tail TTFT (11-52\%) and mean TTFT (6-78\%) across different model-device configurations, while dramatically reducing serving costs by up to 84\% through its migration mechanism while maintaining comparable QoE levels.

Figures (11)

• Figure 1: DiSCo acts as a middleware to optimize QoE by adaptively dispatching and migrating response generation between device and server endpoints under cost constraints.
• Figure 2: On-device TTFT performance is more stable.
• Figure 3: On-device TBT performance is more stable.
• Figure 4: Token generation migration between endpoints. Row A shows the original sequence on the source endpoint, while Row B shows the sequence after migration to the target endpoint, maintaining consistent token delivery while reducing cost.
• Figure 5: Mean TTFT reduction of DiSCo remains significant on DiffusionDB trace.