Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding

TL;DR

This work tackles the high latency of autoregressive LLM inference in edge networks by employing speculative decoding across a heterogeneous edge stack. A pipeline-enabled SD framework splits drafting on a lightweight edge model and verification on a more capable counterpart, enabling multi-token generation per forward pass and substantial latency reductions. The authors derive a comprehensive latency model and present a closed-form wireless-resource allocation plus a dynamic-programming batching/speculation strategy to jointly minimize serving latency under memory and bandwidth constraints. Simulations show the approach outperforms AD-based baselines with up to 44.9% latency reductions across model pairs and task settings, highlighting practical benefits for privacy-preserving, low-latency edge inferencing.

Abstract

Large language model (LLM) inference at the network edge is a promising serving paradigm that leverages distributed edge resources to run inference near users and enhance privacy. Existing edge-based LLM inference systems typically adopt autoregressive decoding (AD), which only generates one token per forward pass. This iterative process, compounded by the limited computational resources of edge nodes, results in high serving latency and constrains the system's ability to support multiple users under growing demands.To address these challenges, we propose a speculative decoding (SD)-based LLM serving framework that deploys small and large models across heterogeneous edge nodes to collaboratively deliver inference services. Specifically, the small model rapidly generates draft tokens that the large model verifies in parallel, enabling multi-token generation per forward pass and thus reducing serving latency. To improve resource utilization of edge nodes, we incorporate pipeline parallelism to overlap drafting and verification across multiple inference tasks. Based on this framework, we analyze and derive a comprehensive latency model incorporating both communication and inference latency. Then, we formulate a joint optimization problem for speculation length, task batching, and wireless communication resource allocation to minimize total serving latency. To address this problem, we derive the closed-form solutions for wireless communication resource allocation, and develop a dynamic programming algorithm for joint batching and speculation control strategies. Experimental results demonstrate that the proposed framework achieves lower serving latency compared to AD-based serving systems. In addition,the proposed joint optimization method delivers up to 44.9% latency reduction compared to benchmark schemes.

Figures (9)

• Figure 1: Illustration of the proposed LLM serving system with speculative decoding.
• Figure 2: The typical workflow of speculative decoding.
• Figure 3: The computation procedure of transformer decoder-only models.
• Figure 4: Illustration of the total latency of pipeline-enabled LLM inference.
• Figure 5: Model runtime of LLaMA-7B versus batch size and FLOPs on NVIDIA RTX4500.

Theorems & Definitions (1)

• Remark 1