Efficient Reasoning for LLMs through Speculative Chain-of-Thought

TL;DR

Speculative Chain-of-Thought (SCoT) tackles latency in large reasoning LLMs by deploying a lightweight draft model to generate multiple CoT drafts in parallel, with a fine-tuned target model selecting the best draft and optionally rethinking for hard cases. Through thinking behavior alignment and a draft-selection+error-correction mechanism, SCoT maintains near-target-model accuracy while delivering substantial speedups (up to $2.92\times$) across diverse benchmarks like GSM8K, MATH, GaoKao, CollegeMath, and Olympiad. The approach demonstrates strong efficiency-accuracy trade-offs, outperforming token-level speculative methods and offering practical benefits for real-world reasoning tasks. Limitations include reliance on fine-tuning data and draft-model choice, with future work aimed at broader task applicability and further robustness improvements.

Abstract

Large reasoning language models such as OpenAI-o1 and Deepseek-R1 have recently attracted widespread attention due to their impressive task-solving abilities. However, the enormous model size and the generation of lengthy thought chains introduce significant reasoning costs and response latency. Existing methods for efficient reasoning mainly focus on reducing the number of model parameters or shortening the chain-of-thought length. In this paper, we introduce Speculative Chain-of-Thought (SCoT), which reduces reasoning latency from another perspective by accelerated average reasoning speed through large and small model collaboration. SCoT conducts thought-level drafting using a lightweight draft model. Then it selects the best CoT draft and corrects the error cases with the target model. The proposed thinking behavior alignment improves the efficiency of drafting and the draft selection strategy maintains the prediction accuracy of the target model for complex tasks. Experimental results on GSM8K, MATH, GaoKao, CollegeMath and Olympiad datasets show that SCoT reduces reasoning latency by 48\%$\sim$66\% and 21\%$\sim$49\% for Deepseek-R1-Distill-Qwen-32B and Deepseek-R1-Distill-Llama-70B while achieving near-target-model-level performance. Our code is available at https://github.com/Jikai0Wang/Speculative_CoT.

Figures (7)

• Figure 1: Average reasoning accuracy and efficiency of Speculative Chain-of-Thought (SCoT) with Deepseek-R1-Distill-Qwen-32B and Deepseek-R1-Distill-Llama-B on GSM8K, MATH, GaoKao, CollegeMath and Olympiad. Graph (a): SCoT achieves near-target-model performance level and outperforms TokenSkip on average generation accuracy on the 5 datasets. Graph (b): SCoT demonstrates an average speed-up ratio of 2.3× and 1.6× for the two target models on reasoning latency. Graph (c): Comparison of reasoning speed and accuracy for models of different sizes. Note that all the experiments for testing the reasoning speed are conducted on H800-80G GPUs. SCoT exhibits a better reasoning efficiency-accuracy trade-off through large and small model collaboration.
• Figure 2: Overview of Speculative Chain-of-Thought. Given a question $Q$, SCoT first applies a lightweight draft model to generate multiple CoT drafts in parallel. The draft model is fine-tuned with LoRA modules to align the thinking behavior of the target model. It appends a special CoT option for the case where all drafts are wrong. The target model for selecting the best CoT draft is also fine-tuned with LoRA modules for improved accuracy. With the designed prompt template $S$, only one forward propagation is needed to get the index of the best CoT. Once the best CoT is picked, SCoT directly adopts it for the original target model to generate the final answer. Only if no draft is selected, SCoT will rethink with the target model to ensure the quality of the generated answer.
• Figure 3: Comparison of reasoning accuracy on GSM8K.
• Figure 4: Comparison of accuracy of the final answer on the 5 datasets with Deepseek-R1-Distill-Qwen (top) and Deepseek-R1-Distill-Llama (bottom). The compression ratio for TokenSkip is set to 0.5. The speed-up ratio in the legends represents the latency speed-up for TokenSkip and SCoT.
• Figure 5: Impact of the chain number on reasoning accuracy and latency on GSM8K.