Traversal Verification for Speculative Tree Decoding

TL;DR

Traversal Verification introduces a leaf-to-root, bottom-up verification strategy for tree-based speculative decoding, enabling sequence-level acceptance and full utilization of drafted tokens. It defines and uses residual distributions $igl[ulletigr]_+$ and modified drafts $igl( ext{M}'_b, ext{M}'_sigr)$ to propagate acceptance from leaves to roots, and proves losslessness: the output distribution matches the target model $igl( ext{M}_b(ullet|X_0)igr)$. The approach also achieves optimality in the single-chain limit, aligning with Block Verification, and empirically improves average acceptance length by 2.2% to 5.7% across multiple tasks and tree structures, with larger gains for deeper and larger trees. Across Llama3 and Llama2 variants on Spec-Bench, Traversal Verification yields meaningful throughput improvements and maintains generation quality, demonstrating practical impact for accelerating LLM inference on diverse workloads.

Abstract

Speculative decoding is a promising approach for accelerating large language models. The primary idea is to use a lightweight draft model to speculate the output of the target model for multiple subsequent timesteps, and then verify them in parallel to determine whether the drafted tokens should be accepted or rejected. To enhance acceptance rates, existing frameworks typically construct token trees containing multiple candidates in each timestep. However, their reliance on token-level verification mechanisms introduces two critical limitations: First, the probability distribution of a sequence differs from that of individual tokens, leading to suboptimal acceptance length. Second, current verification schemes begin from the root node and proceed layer by layer in a top-down manner. Once a parent node is rejected, all its child nodes should be discarded, resulting in inefficient utilization of speculative candidates. This paper introduces Traversal Verification, a novel speculative decoding algorithm that fundamentally rethinks the verification paradigm through leaf-to-root traversal. Our approach considers the acceptance of the entire token sequence from the current node to the root, and preserves potentially valid subsequences that would be prematurely discarded by existing methods. We theoretically prove that the probability distribution obtained through Traversal Verification is identical to that of the target model, guaranteeing lossless inference while achieving substantial acceleration gains. Experimental results across different large language models and multiple tasks show that our method consistently improves acceptance length and throughput over existing methods.

Figures (4)

• Figure 1: An example token tree
• Figure 2: The traversal order of verifying a sampling tree.
• Figure 3: Acceptance lengths and improvements under different chain depths and tree sizes.
• Figure 4: The sequence-level RRSw for two-layers decoding tree.

Theorems & Definitions (11)

• Definition 3.1: Decoding tree under autoregressive generation
• Definition 3.2: Valid tree verification algorithm
• Theorem 3.3: Losslessness of Traversal Verification
• Theorem 3.4
• Lemma A.1
• proof
• Lemma A.2
• proof
• Lemma B.1: Lemma 3 in blockV
• Lemma B.2: Lemma 4 in blockV