StarSD: One-for-Many Speculative Decoding

TL;DR

StarSD tackles inefficiencies of distributed speculative decoding by enabling a single draft model $M_q$ to serve many targets $M_p$ in a star topology, thereby amortizing verification and communication across multiple sessions. The authors develop a system model separating per-iteration progress from time, deriving acceptance probabilities $\beta_i(y)$ and a closed-form for expected accepted tokens $E[l_i(d)]$, yielding throughput $O_\gamma = \mathbb{E}[l]_\gamma / T_\gamma$ with distinct under-loaded and fully-loaded regimes. The design enforces work-conserving draft execution, per-target state isolation, and independent return paths via per-target tags, enabling continuous draft-side throughput and reduced idle gaps. Across intra-node, inter-node, and edge-like deployments, StarSD demonstrates scalable throughput gains, improved draft-side stability, and practical memory packing advantages without altering model weights or generation semantics, while noting areas for fairness and heterogeneity considerations.

Abstract

Speculative decoding accelerates autoregressive generation by separating token proposal from verification, but most existing approaches are designed for single-node execution and do not scale well to multi-accelerator clusters used for serving modern Large Language Models (LLMs). We present StarSD, a one-for-many speculative decoding framework that uses a single draft model to serve multiple target models across distributed nodes via a star topology. StarSD decouples drafting and verification, enabling effective sharing of draft computation, and preventing distributed accelerators from remaining idle under bursty workloads. We provide a system-level analysis that characterizes when and why a single draft model can remain fully utilized by multiple verifiers, yielding predictable latency and utilization gains. Extensive experiments in real-world distributed inference settings demonstrate that StarSD simplifies deployment and supports flexible resource allocation across heterogeneous accelerators, while maintaining output quality. These results indicate that StarSD is a practical and scalable framework for bringing speculative decoding to modern cloud and edge inference infrastructures.

Figures (10)

• Figure 1: Execution cases of StarSD under different load conditions. Fully-loaded case where verification and communication delays of each $M_p^{(i)}$ are overlapped, keeping $M_q$ work-conserving with no idle gaps. Under-loaded case where the draft server idles while waiting for a target to return the next request.
• Figure 2: StarSD Pipeline. Use the global request buffer and outgoing buffer to guarantee the work continuity of the draft model.
• Figure 3: StarSD framework behavior when scaling the number of target instances. Baseline 1 stands for the throughput of SOTA speculative decoding that is deployed on a single accelerator. Baseline 2 is the standalone inference speed of an LLM.
• Figure 4: Throughput loss due to distribute deployment of EAGLE and communication overhead.
• Figure 5: Round speed $R_q$ and $R_p$, load of $M_q$ under varying numbers of target instances $N$.

Theorems & Definitions (4)

• Lemma 3.1: Tail-sum identity for bounded accept length
• proof
• Lemma 3.2: A sufficient non-idling condition for $M_q$
• proof