SpecExit: Accelerating Large Reasoning Model via Speculative Exit

TL;DR

SpecExit addresses the overthinking and high latency of large reasoning models by using a lightweight draft model to predict both future tokens and an early-exit signal directly from hidden states, removing probing overhead. Through multi-task training that augments the MTP layer with signals for confidence, progress, and remaining reasoning length, SpecExit enables dynamic, semantics-aware early termination during decoding. Empirical results across mathematical, scientific, coding, and logical benchmarks show up to 66% reductions in reasoning length and up to 2.5× end-to-end latency improvements with minimal accuracy loss. This approach highlights hidden states as a rich source of efficient reasoning signals and offers a practical deployment pathway for efficient large-scale reasoning systems.

Abstract

Despite their strong performance on reasoning tasks, large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-end latency, a significant limitation to their real-world deployment. To address overthinking, early-exit mechanisms have been proposed to terminate reasoning before typical completion, showing that this approach can effectively shorten generation length with minimal impact on accuracy. However, their reliance on probing mechanisms introduces a detection overhead that limits their end-to-end latency gains and compromises their generalizability across diverse problems. Inspired by the use of hidden states in speculative decoding, we propose SpecExit, a novel framework that predicts both future tokens and an early-exit signal directly from a lightweight draft model without probing overhead. Our method offers significant improvements, reducing average generation length by 66\% and achieving a 2.5x speedup in end-to-end latency compared to the speculative decoding baseline, without compromising accuracy. Our method leverages the inherent signals from hidden states to provide effective early-exit signals, suggesting broader use of hidden states for efficient reasoning. Our code is available at https://github.com/Tencent/AngelSlim.

Figures (10)

• Figure 1: Effectiveness of the proposed method. (a) Statistical comparison showing that our approach produces shorter reasoning chains and faster inference than baselines. (b) visualizes the predicted reasoning progress on a MATH500 example, where darker colors denote insufficient reasoning and lighter colors denote sufficiency, demonstrating valuable signals can be extracted from hidden states regarding the model's reasoning process.
• Figure 2: Overall architecture of the proposed SpecExit framework. The Multi-Token Prediction (MTP) layer is augmented to output both token logits and auxiliary signals. Training is performed with Multi-Task Learning (MTL), while at inference these signals guide speculative early stopping without modifying the backbone model. The example illustrates how redundant reasoning steps can be pruned while preserving final answer quality.
• Figure 3: Convergence of token classification loss and signal regression loss during training.
• Figure 4: Inference process with signal-guided speculative exit.
• Figure 5: Ablation study of SpecExit signal types on Qwen3-4B-Thinking-2507.