Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs

TL;DR

The paper tackles online drafter selection for speculative decoding in LLM inference by reframing the problem as full-information online learning rather than bandit learning. HedgeSpec leverages a between-chunk evaluation phase to obtain feedback for all drafters without extra target-model calls, enabling unbiased estimation of acceptance metrics and no-regret guarantees for TAP or EAL. The authors derive theoretical results, including an unbiased AcceptLength estimator and delayed-feedback regret bounds, and implement a practical Hedge-based algorithm (NormalHedge) that scales to large pools of domain-specialized drafters. Empirically, HedgeSpec consistently outperforms EAGLE3 and BanditSpec across diverse domains and model sizes, achieving substantial gains in token throughput and latency while remaining robust to distribution shifts. These findings demonstrate a scalable, training-free approach to dynamic drafter orchestration with strong theoretical guarantees and real-world applicability for fast, reliable LLM serving.

Abstract

Speculative decoding is widely used in accelerating large language model (LLM) inference. In this work, we focus on the online draft model selection problem in speculative decoding. We design an algorithm that provably competes with the best draft model in hindsight for each query in terms of either the token acceptance probability or expected acceptance length. In particular, we show that we can accurately evaluate all draft models, instead of only the chosen model without incurring additional queries to the target model, which allows us to improve exponentially over the existing bandit-based approach as the number of draft models increases. Our approach is generically applicable with any speculative decoding methods (single draft, multi-drafts and draft-trees). Moreover, we design system-efficient versions of online learners and demonstrate that the overhead in computation and latency can be substantially reduced. We conduct extensive experiments on open-source LLMs and diverse datasets, demonstrating that our methods substantially outperform the state-of-the-art EAGLE3 and the BanditSpec baseline in a variety of domains where specialized domain-expert drafters are available, especially when long reasoning chains are required.

Figures (8)

• Figure 1: Left: Acceptance rates of Qwen-3-8B drafters show strong in-domain performance but sharp degradation outside their expertise, posing challenges for effective drafter selection. HedgeSpec achieves nearly best performance across domains. Right-top: HedgeSpec overall workflow. After the verification phase, a lightweight evaluation collects panoramic feedback from drafters by estimating the distribution gap between each drafter and the target. Right-bottom: Speedup ratio across different mechanisms. HedgeSpec benefits from full-information feedback, resulting in substantial acceleration during inference.
• Figure 2: Illustration of the conditional acceptance probability $\gamma_k$ for the EAGLE model. Observe that $\gamma_k$ is not the probability of accepting $x_k$ only, but rather the total probability of accepting a token at level $x_k$. Also note that for evaluating the estimator, we never need to compute the probabilities of the target model for the possible alternative trajectories that the drafters generate.
• Figure 3: Illustration of the counterfactual feedback for Drafter $j$ (per-token or per-chunk) while the model keeps collecting new data. In this example, the generated token length $K = 5$ For the token-level learner, the feedback is delayed for $K$ steps, while for the chunk-level learner, the delay depends on the average number of chunks to collect $K$-tokens.
• Figure 4: Token-Level Game with Delay.
• Figure 5: Cumulative regret and MAT vs. number of drafters. HedgeSpec quickly settles with near-zero regret, and can scale up with larger drafter pool. Qwen-3-8B results show similar trend in Appendix \ref{['qwen_mat']}, highlighting HedgeSpec's robustness.

Theorems & Definitions (12)

• Theorem 1: leviathan2023fast
• Theorem 2
• Theorem 3
• Theorem 4
• proof
• proof : Proof of Theorem \ref{['thm:unbiased_len_estimate']}
• proof : Proof of Theorem \ref{['thm:regret_token_level']}
• Corollary 5: Regret of QFI-D
• proof
• Theorem 6