Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding

TL;DR

Yggdrasil tackles latency bottlenecks in speculative LLM decoding by co-designing the drafting algorithm with a compile-friendly runtime. It introduces an equal-growth tree to maintain static operator shapes while remaining adaptive to context and a latency-aware objective that better reflects real hardware costs. A stage-based scheduling runtime further reduces CPU-GPU overhead by overlapping and prefetching speculative stages. Across diverse models and hardware, Yggdrasil achieves substantial end-to-end speedups over state-of-the-art baselines, validating the benefits of integrating algorithmic drafting with compiler-aware execution.

Abstract

Speculative decoding improves LLM inference by generating and verifying multiple tokens in parallel, but existing systems suffer from suboptimal performance due to a mismatch between dynamic speculation and static runtime assumptions. We present Yggdrasil, a co-designed system that enables latency-optimal speculative decoding through context-aware tree drafting and compiler-friendly execution. Yggdrasil introduces an equal-growth tree structure for static graph compatibility, a latency-aware optimization objective for draft selection, and stage-based scheduling to reduce overhead. Yggdrasil supports unmodified LLMs and achieves up to $3.98\times$ speedup over state-of-the-art baselines across multiple hardware setups.

Figures (15)

• Figure 1: Overview of Yggdrasil.
• Figure 2: Illustration of speculative decoding.
• Figure 3: Comparison of drafting structures.
• Figure 4: Benchmark of different runtimes.
• Figure 5: Latency and speedup characteristics.