WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching

TL;DR

This paper tackles the challenge of scalable LLM inference at the edge and cloud by introducing WISP, a system that co-designs drafting control, SLO-aware batching, and a verification-time estimator to address two bottlenecks: Wasted Drafting Time (WDT) and Verification Interference. The approach includes (i) an intelligent drafting controller using a lightweight rejection predictor to stop drafting early, (ii) an SLO-aware batch scheduler framed as a knapsack-like problem with a fast path for urgent requests, and (iii) a fine-grained additive verification-time model to predict batch latency. Key findings show that WISP can improve system capacity by up to 2.1× over centralized serving and 4.1× over SLED, and increase goodput by up to 1.94× and 3.7× respectively, while reducing token-speed SLO violations under heterogeneous interference. The results demonstrate practical gains for edge-enabled interactive LLM services, enabling higher utilization of edge devices and centralized GPUs in a coordinated, latency-aware manner.

Abstract

As Large Language Models (LLMs) become increasingly accessible to end users, an ever-growing number of inference requests are initiated from edge devices and computed on centralized GPU clusters. However, the resulting exponential growth in computation workload is placing significant strain on data centers, while edge devices remain largely underutilized, leading to imbalanced workloads and resource inefficiency across the network. Integrating edge devices into the LLM inference process via speculative decoding helps balance the workload between the edge and the cloud, while maintaining lossless prediction accuracy. In this paper, we identify and formalize two critical bottlenecks that limit the efficiency and scalability of distributed speculative LLM serving: Wasted Drafting Time and Verification Interference. To address these challenges, we propose WISP, an efficient and SLO-aware distributed LLM inference system that consists of an intelligent speculation controller, a verification time estimator, and a verification batch scheduler. These components collaboratively enhance drafting efficiency and optimize verification request scheduling on the server. Extensive numerical results show that WISP improves system capacity by up to 2.1x and 4.1x, and increases system goodput by up to 1.94x and 3.7x, compared to centralized serving and SLED, respectively.

Figures (11)

• Figure 1: Impact of WDT on Device Goodput
• Figure 2: Relationships of Draft Logits Statistics with Acceptance Rate
• Figure 3: Correlation between Logits Statistics and Token acceptance
• Figure 4: Verification Interference among Heterogeneous Requests
• Figure 5: WISP architecture and data flow

Theorems & Definitions (1)

• Theorem 1: Monotonicity of WDT