Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match

TL;DR

The paper tackles latency in autoregressive LLM decoding by relaxing SPD's exact-match verification through Training-Free Loosely Speculative Decoding (FLy). FLy uses an entropy-level gate and a token-level deferred window to accept semantically valid but non-identical draft tokens, leveraging the target model's self-corrective behavior. It also introduces multi-level acceleration to speed up both the drafter and the target, all without training. Experiments show FLy achieves substantial end-to-end speedups (often severalfold) with over 99% recovery of the target's accuracy on both in-domain and out-of-domain tasks, and it generalizes across model sizes and draft-target pairings.

Abstract

Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SPD) mitigates this issue by verifying candidate tokens in parallel from a smaller draft model, yet its strict exact-match verification discards many semantically valid continuations. Moreover, existing training-based SPD methods often suffer from performance degradation on out-of-distribution (OOD) tasks. To this end, we propose Training-Free Loosely Speculative Decoding (FLy), a novel method that loosens the rigid verification criterion by leveraging the target model's self-corrective behavior to judge whether a draft-target mismatch remains semantically valid. FLy introduces a two-tier mechanism: an entropy-level gate that identifies whether the current token allows multiple plausible alternatives or is nearly deterministic, and a token-level deferred window that distinguishes genuine errors from differently worded yet semantically correct variants. To further reduce latency, we design a multi-level acceleration strategy that accelerates not only the target model but also the drafter itself. Owing to its training-free design, FLy composes seamlessly with arbitrary draft-target pairs and generalizes across models and domains without hyperparameter re-tuning. Experiments show that FLy preserves more than 99% of the target model's accuracy while achieving an average 2.81x speedup on Llama-3.1-70B-Instruct and 5.07x speedup on the 405B variant. Notably, on out-of-domain datasets, our method remains highly effective and outperforms the training-based method EAGLE-3 by 1.62x.

Figures (4)

• Figure 1: Speedup on out-of-domain (OOD) datasets. Training-based method EAGLE-3 suffers significant degradation under OOD conditions. Our approach surpasses existing methods across datasets and models by accepting more semantically valid tokens, achieving SOTA performance.
• Figure 2: Overview of our proposed FLy. (1) When the draft and target tokens differ, we do not immediately reject the case as in prior SPD methods. Instead, our two-tier scheme assesses whether the mismatch is semantically valid and rejects only truly invalid cases. (2) An entropy gate rejects deterministic target predictions where $h<\theta$, deferring ambiguous mismatches. (3) A token-level deferral window ($W=6$) then monitors for continued divergence. (4) The final generation demonstrates that FLy admits more semantically valid continuations, whereas standard SPD would reject at the first mismatch.
• Figure 3: Accuracy preservation results. The performance of the target model is normalized to 100, and the recovery ratio is used to quantify the extent of performance preservation.
• Figure 4: Case study of FLy on a sample from the GSM8K dataset using Llama-3.1-405B-Instruct. Blue tokens denote the drafter’s output and green tokens denote the target’s output. Under standard speculative decoding, all mismatches would be rejected. In contrast, FLy selectively rejects only the mismatches shown with strikethrough, while retaining the remaining ones because they are semantically valid.