ConfSpec: Efficient Step-Level Speculative Reasoning via Confidence-Gated Verification

TL;DR

ConfSpec tackles the latency of step-level chain-of-thought reasoning in large language models by introducing a confidence-gated cascaded verification framework that delegates uncertain steps to a high-capacity target model while accepting high-confidence draft decisions. By leveraging an empirical verification asymmetry and the calibration of draft models within their competence range, ConfSpec achieves substantial end-to-end speedups (up to $2.24\times$) while preserving target-model accuracy and without external judges. The approach is compatible with token-level speculative decoding, enabling multiplicative acceleration, and demonstrates robustness across mathematical, scientific, and code-generation tasks. Overall, ConfSpec provides a principled solution to the accuracy-speed-resource trade-off in step-level speculative reasoning with practical deployment benefits.

Abstract

Chain-of-Thought reasoning significantly improves the performance of large language models on complex tasks, but incurs high inference latency due to long generation traces. Step-level speculative reasoning aims to mitigate this cost, yet existing approaches face a long-standing trade-off among accuracy, inference speed, and resource efficiency. We propose ConfSpec, a confidence-gated cascaded verification framework that resolves this trade-off. Our key insight is an asymmetry between generation and verification: while generating a correct reasoning step requires substantial model capacity, step-level verification is a constrained discriminative task for which small draft models are well-calibrated within their competence range, enabling high-confidence draft decisions to be accepted directly while selectively escalating uncertain cases to the large target model. Evaluation across diverse workloads shows that ConfSpec achieves up to 2.24$\times$ end-to-end speedups while matching target-model accuracy. Our method requires no external judge models and is orthogonal to token-level speculative decoding, enabling further multiplicative acceleration.

Figures (1)

• Figure 1: Confidence-based filtering improves verification accuracy. (a) Restricting to high-confidence predictions ($p_D \ge 0.9$) substantially improves draft-model accuracy, approaching target-model quality. (b) High-confidence cases cover the majority of verification instances (61--85%), enabling efficient cascading.