Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits

TL;DR

The paper addresses the high cost and latency of deploying large LLMs by proposing a speculative edge-cloud decoding framework that allocates the draft model to edge devices and the target model to servers, augmented with early exits for preemptive drafting. This design enables tokens to be drafted while verification occurs, significantly increasing parallelism and reducing idle time. Empirical results show up to a 35% latency reduction over cloud autoregressive decoding and an extra 11% gain from pre-drafting, with a real-world robotics demonstration achieving 21% speedup on the Unitree Go2. The approach is validated across multiple model pairings and tasks, indicating practical viability for real-time edge-LM and VLM tasks in resource-constrained environments. The work also introduces training-time adapters for early exits and analyzes system-level trade-offs, offering a pathway toward scalable edge-cloud collaborative inference.

Abstract

Large Language Models (LLMs) enable various applications on edge devices such as smartphones, wearables, and embodied robots. However, their deployment often depends on expensive cloud-based APIs, creating high operational costs, which limit access for smaller organizations and raise sustainability concerns. Certain LLMs can be deployed on-device, offering a cost-effective solution with reduced latency and improved privacy. Yet, limited computing resources constrain the size and accuracy of models that can be deployed, necessitating a collaborative design between edge and cloud. We propose a fast and cost-effective speculative edge-cloud decoding framework with a large target model on the server and a small draft model on the device. By introducing early exits in the target model, tokens are generated mid-verification, allowing the client to preemptively draft subsequent tokens before final verification, thus utilizing idle time and enhancing parallelism between edge and cloud. Using an NVIDIA Jetson Nano (client) and an A100 GPU (server) with Vicuna-68M (draft) and Llama2-7B (target) models, our method achieves up to a 35% reduction in latency compared to cloud-based autoregressive decoding, with an additional 11% improvement from preemptive drafting. To demonstrate real-world applicability, we deploy our method on the Unitree Go2 quadruped robot using Vision-Language Model (VLM) based control, achieving a 21% speedup over traditional cloud-based autoregressive decoding. These results demonstrate the potential of our framework for real-time LLM and VLM applications on resource-constrained edge devices.

Figures (6)

• Figure 1: Illustration of traditional cloud-based autoregressive decoding versus cloud-based speculative decoding, vanilla speculative edge-cloud decoding, and the proposed preemptive drafting mechanism.
• Figure 2: Illustration of our proposed approach. Given a prefix, the client generates two draft tokens and sends them to the server. The server verifies them using a target model with early exits, returning verified tokens and the next generated token. For each early exit, the client pre-drafts the next tokens and stores them in the pre-draft cache. If the final output matches a cache entry, the draft tokens are sent immediately, reducing latency.
• Figure 3: Illustration of training early exit adapters.
• Figure 4: Ablation studies: (a) Effect of varying the number of early exit threads, (b) Effect of varying the number of draft tokens ($\gamma$).
• Figure 5: Example run of the Unitree Go2 robot performing an object-finding task using vision-language-based control. The robot receives the instruction “find the silver bottle” and navigates the environment while distinguishing the correct object from a similar distractor.