SpecAttn: Co-Designing Sparse Attention with Self-Speculative Decoding

TL;DR

Long-context LLM inference is bottlenecked by KV cache memory bandwidth. SpecAttn co-designs drafting and verification by using full-attention results from the verification pass to identify critical KV entries and guide sparse attention during drafting, complemented by low-overhead logit collection via Collect-2-Query. The method is training-free and patch-friendly for vLLM, achieving $2.81\times$ throughput over vanilla decoding and up to $1.29\times$ over state-of-the-art baselines across diverse models and long-context tasks. This approach enables high drafting accuracy with minimal KV-selection overhead, delivering significant practical speedups without sacrificing output quality.

Abstract

Long-context large language model (LLM) inference has become the norm for today's AI applications. However, it is severely bottlenecked by the increasing memory demands of its KV cache. Previous works have shown that self-speculative decoding with sparse attention, where tokens are drafted using a subset of the KV cache and verified in parallel with full KV cache, speeds up inference in a lossless way. However, this approach relies on standalone KV selection algorithms to select the KV entries used for drafting and overlooks that the criticality of each KV entry is inherently computed during verification. In this paper, we propose SpecAttn, a self-speculative decoding method with verification-guided sparse attention. SpecAttn identifies critical KV entries as a byproduct of verification and only loads these entries when drafting subsequent tokens. This not only improves draft token acceptance rate but also incurs low KV selection overhead, thereby improving decoding throughput. SpecAttn achieves 2.81$\times$ higher throughput over vanilla auto-regressive decoding and 1.29$\times$ improvement over state-of-the-art sparsity-based self-speculative decoding methods.

Figures (12)

• Figure 1: Comparison of (a) existing self-speculative decoding with sparse attention and (b) self-speculative decoding with verification-guided sparse attention (). Each subfigure shows one decoding iteration (drafting + verification). improves decoding throughput by identifying critical KV entries as a byproduct of verification and applying sparse attention over these entries when drafting subsequent tokens, enabling a high draft token acceptance rate. Note that details such as the bonus token are omitted for simplicity.
• Figure 2: The average number of accepted draft tokens (excluding the bonus token) per decode iteration using StreamingLLM and Quest as drafters in self-speculative decoding. Quest achieves higher drafting accuracy via its query-aware attention sparsity compared to the query-agnostic StreamingLLM. Results are profiled on LongBench-v2 with 7% of KV cache entries selected.
• Figure 3: Execution time breakdown of one decoding iteration (drafting 5 tokens + verification) using StreamingLLM and Quest as drafters. Results are profiled on LongBench-v2 when running Qwen3-8B with a batch size of 4 on an H100 GPU.
• Figure 4: Comparison of two KV selection strategies. Selecting KV entries with the highest attention weights for the last accepted token results in a rapid decay in acceptance rate as the draft token's position in the draft chain increases. In contrast, selecting KV entries that maximize overall attention weight coverage across all draft tokens maintains consistently high acceptance rates. The y-axis starts at 0.85 for clarity. Results are profiled on LongBench-v2 with 7% of KV entries selected.
• Figure 5: Distribution of acceptance rates of rejected tokens. Many rejected tokens are sampled from a distribution close to the distribution predicted by the full attention.