EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization

TL;DR

<3-5 sentence high-level summary> EasySpec addresses underutilization in multi-GPU speculative decoding by introducing layer-parallel, fuzzy speculation that breaks inter-layer dependencies in the draft model. It couples this with a bonus calibration step that refreshes precise KV caches without substantial latency, making the drafting stage faster while preserving the base-model output distribution. The method is training-free and plug-in, achieving up to 4.17x peak speedup and up to 1.62x drafting acceleration with limited accuracy loss. Across multiple open-source LLMs and tasks, EasySpec demonstrates robust improvements in throughput with maintained stability and generalization.

Abstract

Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, a layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependencies in the draft model, enabling multiple layers to run simultaneously across multiple devices as 'fuzzy' speculation. After each drafting-and-verification iteration, the draft model's key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation at minimal additional latency. EasySpec is a training-free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve a peak speedup of 4.17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with a maximum speculation accuracy drop of only 7%. The code is available at https://github.com/Yize-Wu/EasySpec.

Figures (6)

• Figure 1: Overview of EasySpec: comparison between standard Speculative Decoding and EasySpec. The differences are mainly about: (1) Fuzzy Speculation uses layer parallelization for drafting acceleration. (2) The fuzzy KV items are discarded regardless of acceptance length, while Speculative Decoding preserves accepted items. (3) Speculative Decoding is 2-staged and EasySpec is 3-staged, with an additional stage for bonus calibration.
• Figure 2: GPU runtime of different decoding algorithms in the distributed system. Speculative Decoding (SD) enables token-level parallelism (fewer base-model blocks), while causing multi-GPU under-utilization during the drafting stage. EasySpec solves the problem by layer-parallel speculation.
• Figure 3: 2-D demonstration of high cosine similarity between $h_i$ and $h_{i+1}$.
• Figure 4: Token throughput and acceptance rates of Llama-3-70B(8B)-Instruct on MMLU under different configurations of EasySpec. Left: token throughput. Right: acceptance rates.
• Figure 5: Illustration of tree attention. 6 draft token sequences are generated within 2 forward passes, which increases parallelism and potentially acceptance rates. The whole tree can be verified in a single-forward pass of the base model.