Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput

TL;DR

Speculative Decoding in Decentralized LLM Inference targets the problem of high inter-node latency in distributed LLM serving. The authors introduce Decentralized Speculative Decoding (DSD), a training-free framework that verifies $k$ tokens in a single synchronization round across $N$ nodes, turning communication delays into computation with a throughput gain described by $T_{std} = k(t_0 + (N-1)t_1)$ and $T_{DSD} = k t_0 + (N-1)t_1$, yielding a comms reduction $R_{comm} = \frac{(N-1)t_1 (k-1)}{k (t_0 + (N-1)t_1)}$. A key contribution is adaptive speculative verification, which uses token-level semantic importance to relax acceptance for low-impact tokens while strictly verifying high-impact ones, controlled by a relaxation parameter $\tau$. Empirically, DSD achieves up to $2.56\times$ speedup on HumanEval and $2.59\times$ on GSM8K with accuracy preserved, and reduces inter-node communication by about $37\%$ at eight nodes, outperforming Eagle3 without retraining. The work demonstrates that coordinating speculative decoding with decentralized execution can dramatically improve system throughput for distributed LLM inference, with practical impact for federated and cross-region deployments.

Abstract

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens that are later verified by a stronger target model. While effective in centralized systems, its behavior in decentralized settings, where network latency often dominates compute, remains under-characterized. We present Decentralized Speculative Decoding (DSD), a plug-and-play framework for decentralized inference that turns communication delay into useful computation by verifying multiple candidate tokens in parallel across distributed nodes. We further introduce an adaptive speculative verification strategy that adjusts acceptance thresholds by token-level semantic importance, delivering an additional 15% to 20% end-to-end speedup without retraining. In theory, DSD reduces cross-node communication cost by approximately (N-1)t1(k-1)/k, where t1 is per-link latency and k is the average number of tokens accepted per round. In practice, DSD achieves up to 2.56x speedup on HumanEval and 2.59x on GSM8K, surpassing the Eagle3 baseline while preserving accuracy. These results show that adapting speculative decoding for decentralized execution provides a system-level optimization that converts network stalls into throughput, enabling faster distributed LLM inference with no model retraining or architectural changes.

Figures (2)

• Figure 1: Roofline view of attainable performance versus arithmetic intensity. Prefill is compute oriented at moderate batch size, while token by token decoding is often memory bound dao2022flashattention. Verifying a compact draft window increases effective intensity.
• Figure 2: Decentralized speculative decoding. Left: model shards on distributed nodes. Right: a draft window is verified; if the first $k$ tokens pass, they are committed together in one round, improving pipeline utilization.