Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding

TL;DR

The paper addresses the memory-bound bottleneck in large reasoning language model inference caused by KV-Cache growth during long generations.It introduces SparseSpec, a training-free self-speculative decoding framework featuring PillarAttn for dynamic sparse attention and a co-designed algorithm-system stack (unified scheduler, delayed verification, dynamic KV-Cache management) to accelerate end-to-end inference.Empirical results across multiple Qwen-3 models and reasoning benchmarks show up to 2.13x throughput gains over state-of-the-art baselines and strong improvements over other speculative decoding approaches, without additional training.The work demonstrates practical potential for deploying efficient long-output reasoning systems and outlines avenues for extension to mixture-of-experts models and hierarchical speculative strategies.

Abstract

Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth. To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup.

Figures (15)

• Figure 1: Comparison between autoregressive generation (a), draft model-based speculative decoding (b), and SparseSpec. SparseSpec identifies the KV-Cache loading as the key bottleneck during token generation, and uses the same model weights with dynamic sparse attention as the draft model to achieve high efficiency and a high acceptance rate without additional training.
• Figure 2: Compute and memory bandwidth utilization of Qwen3-8B on an H100, with an average output of $12$K on AIME.
• Figure 3: Theoretical and achieved speedup over vLLM of MagicDec and self-speculation with oracle Top-K attention. Assume a sparsity ratio $s=0.5$ and speculative step $k=8$.
• Figure 4: Visualization of attention scores of Qwen3-8B on the AIME dataset. While the attention pattern has spatial locality, it undergoes substantial changes during generation.
• Figure 5: The memory utilization and recomputation ratio during the first $100$K steps when serving Qwen3-8B with AIME on H100. Existing methods either underutilizes KV-Cache capacity, or leads to excessive recomputation. Our dynamic KV-Cache manager can fully utilize the capacity without incurring recomputation.