Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

Bradley McDanel, Sai Qian Zhang, Yunhai Hu, Zining Liu

TL;DR

PipeSpec addresses the slow decode phase in autoregressive LLMs by introducing a hierarchical, asynchronous pipeline of multiple draft and verifier models, enabling continuous parallel execution. The authors develop an analytical model that characterizes token generation rates and steady-state verification probabilities, proving throughput improvements over autoregressive decoding for any non-zero acceptance rate. Empirical results show up to 2.54x speedups on modern LLM variants (e.g., LLaMA3.1-70B) with deeper pipelines, with performance increasing as pipeline depth grows. The framework demonstrates scalable acceleration on multi-device systems for tasks like text summarization and code generation, while highlighting practical considerations such as static configurations and rollback costs.

Abstract

Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to $k$ models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54$\times$ speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.

PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

TL;DR

PipeSpec addresses the slow decode phase in autoregressive LLMs by introducing a hierarchical, asynchronous pipeline of multiple draft and verifier models, enabling continuous parallel execution. The authors develop an analytical model that characterizes token generation rates and steady-state verification probabilities, proving throughput improvements over autoregressive decoding for any non-zero acceptance rate. Empirical results show up to 2.54x speedups on modern LLM variants (e.g., LLaMA3.1-70B) with deeper pipelines, with performance increasing as pipeline depth grows. The framework demonstrates scalable acceleration on multi-device systems for tasks like text summarization and code generation, while highlighting practical considerations such as static configurations and rollback costs.

Abstract

Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54 speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.
Paper Structure (15 sections, 1 theorem, 6 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 15 sections, 1 theorem, 6 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Inference Acceleration
  4. Speculative Decoding
  5. Pipelined Speculative Decoding
  6. Overview
  7. Algorithm
  8. Theoretical Performance Analysis
  9. Evaluation
  10. Experimental Setup
  11. Performance Analysis
  12. Token Generation Distribution and Timing Analysis
  13. Token Lookahead Analysis
  14. Resource Utilization
  15. Conclusion

Key Result

Theorem 1

For any $0 < \alpha < 1$ and $0 < \gamma$, the PipeSpec scheme generates a higher number of tokens per step.

Figures (5)

  • Figure 1: Comparison of different LLM decoding approaches. Top Left: Traditional autoregressive decoding (1 token/unit). Top Right: Speculative decoding using a small draft model (10 tokens/unit) for parallel verification by a large model (1.5 tokens/unit). Bottom: Our PipeSpec framework with $k-1$ draft models in a pipeline feeding into the large model ($M_k$), achieving 2.25 tokens/unit through pipelined parallelism. Checkmarks (✓) show accepted predictions while crosses (✗) indicate rejections triggering pipeline rollbacks.
  • Figure 2: Comparison of different decoding approaches showing token generation over time. From top to bottom: (1) Traditional autoregressive decoding (AR) with sequential token generation using a single model $M_2$, (2) Standard speculative decoding (SD) using a draft model $M_0$ to generate candidate tokens verified in batches by $M_2$, (3) PipeSpec (PS) with 2 models showing continuous parallel execution between $M_0$ and $M_2$, and (4) PipeSpec with 3 models demonstrating hierarchical speculation across {$M_0$, $M_1$, $M_2$}.
  • Figure 3: Analysis of token acceptance patterns and timing across decoding strategies on HumanEval. Top: Distribution of accepted tokens per verify step, showing SD's fixed window behavior versus PipeSpec's more flexible patterns. Bottom: Average time per token as a function of batch size, demonstrating PipeSpec's minimal synchronization overhead.
  • Figure 4: Impact of lookahead window size on token generation time. SD shows poor performance at small windows due to synchronization overhead and at large windows due to wasted speculation. PS maintains lower latency at small windows but degrades at larger sizes as verification must wait for draft tokens.
  • Figure 5: GPU utilization over time showing autoregressive (70B model split across 2 GPUs), speculative decoding ({1B,8B,70B}), and PipeSpec ({1B,8B,70B}). PipeSpec achieves higher average utilization (39.7%) by eliminating idle periods between draft and verification.

Theorems & Definitions (1)

  • Theorem 1