Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Confidence Estimates Decide When Chain-of-Thought Is Necessary for LLMs?

Samuel Lewis-Lim, Xingwei Tan, Zhixue Zhao, Nikolaos Aletras

TL;DR

Chain-of-thought (CoT) prompting boosts reasoning but raises token costs and latency. The paper introduces confidence-gated CoT, activating CoT only when a direct answer is insufficient, using four training-free signals: perplexity, $P(\text{True})$, margin sampling, and verbalised confidence. Across seven datasets and multiple models, these signals reduce redundant CoT usage by about 25–30% while largely preserving accuracy on larger models, though performance is inconsistent across tasks and calibrations. The study highlights both the promise and limitations of current confidence-estimation methods for adaptive CoT, pointing to calibration and reliability as critical directions for future work.

Abstract

Chain-of-thought (CoT) prompting has emerged as a common technique for enhancing the reasoning abilities of large language models (LLMs). While extended reasoning can boost accuracy on complex tasks, it is often unnecessary and substantially increases token usage, limiting the practicality of reasoning models in many scenarios. Recent models, such as GPT-OSS and Qwen3, expose controls that enable users to adjust the length of CoT or determine whether it is used at all. Yet, it remains unclear when CoT should be used: on some tasks it improves performance, while on others it provides little benefit or even harms performance. We address this challenge with confidence-gated CoT, where a model invokes reasoning only when confidence in its direct answer is low. To this end, we present the first systematic study of training-free confidence estimation methods for CoT gating. Specifically, we evaluate four training-free confidence estimation methods and compare them to a random baseline and an oracle that always knows when CoT is needed. Through extensive experiments, we show that existing training-free confidence measures can reduce redundant CoT and outperform randomly invoked CoT. However, the utility of individual confidence measures is inconsistent, varying with both the dataset and the model, underscoring the difficulty of deploying confidence-gated CoT in practice. By analysing both strengths and failure modes, our study highlights the potential and limitations of current methods and paves the way toward more reliable adaptive gating of CoT.

Can Confidence Estimates Decide When Chain-of-Thought Is Necessary for LLMs?

TL;DR

Chain-of-thought (CoT) prompting boosts reasoning but raises token costs and latency. The paper introduces confidence-gated CoT, activating CoT only when a direct answer is insufficient, using four training-free signals: perplexity, , margin sampling, and verbalised confidence. Across seven datasets and multiple models, these signals reduce redundant CoT usage by about 25–30% while largely preserving accuracy on larger models, though performance is inconsistent across tasks and calibrations. The study highlights both the promise and limitations of current confidence-estimation methods for adaptive CoT, pointing to calibration and reliability as critical directions for future work.

Abstract

Chain-of-thought (CoT) prompting has emerged as a common technique for enhancing the reasoning abilities of large language models (LLMs). While extended reasoning can boost accuracy on complex tasks, it is often unnecessary and substantially increases token usage, limiting the practicality of reasoning models in many scenarios. Recent models, such as GPT-OSS and Qwen3, expose controls that enable users to adjust the length of CoT or determine whether it is used at all. Yet, it remains unclear when CoT should be used: on some tasks it improves performance, while on others it provides little benefit or even harms performance. We address this challenge with confidence-gated CoT, where a model invokes reasoning only when confidence in its direct answer is low. To this end, we present the first systematic study of training-free confidence estimation methods for CoT gating. Specifically, we evaluate four training-free confidence estimation methods and compare them to a random baseline and an oracle that always knows when CoT is needed. Through extensive experiments, we show that existing training-free confidence measures can reduce redundant CoT and outperform randomly invoked CoT. However, the utility of individual confidence measures is inconsistent, varying with both the dataset and the model, underscoring the difficulty of deploying confidence-gated CoT in practice. By analysing both strengths and failure modes, our study highlights the potential and limitations of current methods and paves the way toward more reliable adaptive gating of CoT.
Paper Structure (42 sections, 3 equations, 15 figures, 3 tables)

This paper contains 42 sections, 3 equations, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adaptive Reasoning
  4. Model Cascades and Routing
  5. Confidence and Uncertainty Estimation in LLMs
  6. Confidence-Gated Chain-of-Thought
  7. Problem definition
  8. Chain-of-Thought:
  9. Direct:
  10. Self-Assessed Confidence
  11. Perplexity:
  12. $P($True$)$ kadavath2022languagemodelsmostlyknow:
  13. Margin Sampling:
  14. Verbalised Confidence:
  15. Budgets and Pareto-Optimal Thresholds
  16. ...and 27 more sections

Figures (15)

  • Figure 1: Confidence-gated CoT controls if a query is answered directly or with reasoning: high-confidence queries are answered directly, while low-confidence ones trigger reasoning.
  • Figure 2: Offline accuracy–efficiency trade-offs under percentile budgets. Accuracy vs. CoT usage (top) and vs. average tokens (bottom), aggregated over all datasets for GPT-OSS-20B (medium effort), Qwen3-32B, and Qwen3-8B. Curves show verbalised, perplexity, $P(\text{True})$, and margin vs. the random baseline; stars denote the oracle. Full GPT-OSS results for low/medium/high effort are in Appendix \ref{['app:all_dataset-tradeoffs']}.
  • Figure 3: Task-level accuracy–efficiency trade-offs. Representative datasets (CSQA, StrategyQA, GSM8K, GPQA) comparing confidence-gating to random and oracle across models.
  • Figure 4: Reliability diagram for GPT-OSS-20B with margin confidence.
  • Figure 5: Online Accuracy vs. CoT deferral rate (top) and average tokens (bottom) across all datasets in the online setting. Stars show oracle performance.
  • ...and 10 more figures