MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification

TL;DR

This work tackles inference latency in autoregressive LLMs by addressing inefficiencies in speculative decoding's verification step. It introduces Margin-Aware Speculative Verification (MARS), a training-free, plug-and-play strategy that adapts verification to the target model's local decisiveness using the logit margin, accepting plausible runner-up tokens in low-margin regimes. Across 8B–235B models and diverse benchmarks, MARS delivers substantial end-to-end speedups (e.g., up to $\4.76\times$) while preserving generation quality to near-baseline levels (recovery ~100%). The findings show that margin-aware, adaptive verification is a practical, scalable improvement for lossy speculative decoding, enabling faster, reliable LLM inference in real-world settings.

Abstract

Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by tightly coupling the drafter with the target model, the verification mechanism itself remains largely unchanged, relying on strict token-level rejection sampling. In practice, modern LLMs frequently operate in low-margin regimes where the target model exhibits weak preference among top candidates. In such cases, rejecting plausible runner-up tokens yields negligible information gain while incurring substantial rollback cost, leading to a fundamental inefficiency in verification. We propose Margin-Aware Speculative Verification, a training-free and domain-agnostic verification strategy that adapts to the target model's local decisiveness. Our method conditions verification on decision stability measured directly from the target logits and relaxes rejection only when strict verification provides minimal benefit. Importantly, the approach modifies only the verification rule and is fully compatible with existing target-coupled speculative decoding frameworks. Extensive experiments across model scales ranging from 8B to 235B demonstrate that our method delivers consistent and significant inference speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks.

Figures (5)

• Figure 1: Comparison of Logit Ratio vs. Probability Ratio for Adaptive Verification (Qwen3-8B). Red points denote tokens accepted by our relaxation strategy (Logit Ratio $> 0.9$). (a) Top-2 Logits: Relaxed tokens cluster along the diagonal ($z_2 \approx z_1$), indicating our method captures candidates with similar raw scores regardless of scale. (b) Top-2 Probabilities: Unlike probability-based metrics, logit-based relaxation is not confined to high-entropy regions ($p_1 \approx p_2$); it also captures candidates with relatively lower probability ratios. (c) Metric Decoupling: High logit ratios (red) do not imply high probability ratios. The red points span the full range of probability ratios (y-axis), demonstrating that our method recovers valid candidates that are otherwise suppressed by the exponential sensitivity of the softmax function. (d) Relaxation Zone Effectiveness: A zoom-in on the low-margin regime (Logit Ratio $> 0.9$), where red points denote rejected tokens and purple points denote accepted ones.
• Figure 2: Overview of the MARS Verification Workflow. We illustrate the adaptive decision mechanism with a threshold $\theta=0.9$. (Left) Low-Margin Regime: The target model exhibits weak preference between the top candidates "nice" and "lovely" (Logit Ratio $r = 0.911 > \theta$). Identifying this local indifference, MARS accepts the draft token "lovely" (Top-2) as a valid tie-breaker, avoiding unnecessary rollback. (Right) High-Margin Regime: The target model decisively prefers "walk" over the runner-up "work" (Logit Ratio $r = 0.728 < \theta$). MARS detects this stability and reverts to strict verification, rejecting the draft to preserve generation fidelity.
• Figure 3: Accuracy preservation (recovery) across tasks and target models. For each task, we normalize the target model's accuracy to 100 and report the recovery ratio (i.e., our accuracy divided by the target accuracy, in %), which quantifies how much of the target performance is preserved under our (lossy) SD decoding.
• Figure 4: Effect of the logit ratio threshold $\theta$ on quality--efficiency trade-off. We sweep $\theta \in \{0.84, 0.86, 0.88, 0.90, 0.92, 0.94, 0.96\}$ and report accuracy (blue, left axis) and speedup (purple, right axis) on HumanEval and GSM8K with $K\in\{7,10\}$. Increasing $\theta$ consistently reduces speedup, while accuracy typically peaks around $\theta\approx0.90$, indicating a balanced default choice.
• Figure 5: Histograms of top-2 statistics on Qwen3-8B. (a) Distribution of the top-1 logit values; the red dashed line marks $x{=}0$ (0.0% of top-1 logits are negative). (b) Distribution of logit ratios between the 2nd and 1st candidates ($z_2/z_1$); the red dashed line indicates the threshold ($0.9$). (c) Distribution of probability ratios between the 2nd and 1st candidates ($p_2/p_1$); the red dashed line indicates the same threshold ($0.9$).