Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

TL;DR

SemanticSpec reframes speculative decoding from token-level to semantic-level by drafting and verifying semantic sequences rather than individual tokens. It introduces a semantic probability predictor trained offline by probing internal hidden states to estimate the likelihood of expressing a given meaning, enabling semantic-aware verification during decoding. Online verification compares draft and target semantic probabilities and accepts drafts according to their agreement, improving efficiency while preserving fidelity. Across four benchmarks and two model pairs, SemanticSpec achieves substantial speedups (up to 2.7x) and outperforms both token-level and sequence-level baselines in effectiveness and throughput.

Abstract

Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings.Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness.

Figures (5)

• Figure 1: semantic-aware speculative decoding. Instead of drafting and verifying token-by-token, semantic-aware speculative decoding drafts and verifying sequence-by-sequence.
• Figure 2: The distribution of hidden states with a various range of semantic probability in two-dimension. For each 0.1 probability interval, we randomly sample hidden state vectors, then project them into two dimensions using UMAP.
• Figure 3: Comparison of pass@1 and latency across different speculative inference schemes and model pairs. Each point represents a decoding approach. The top row shows results for the target model DeepSeekR1-32B, and the bottom row for QwQ-32B.
• Figure 4: Performance comparison under different size of drafted sequence ($\gamma = {1,2,3}$).
• Figure 5: The impact of different splitting strategies ours vs. sentence-ending punctuation.