No Global Plan in Chain-of-Thought: Uncover the Latent Planning Horizon of LLMs

TL;DR

This work investigates how large language models internally plan during Chain-of-Thought reasoning. By introducing Tele-Lens, a low-rank adapter-based probe, the authors reveal a predominantly myopic planning horizon where precise global planning rarely emerges before CoT ends, especially in explicit compositional tasks. They show that focusing on a small set of pivot positions within CoT can meaningfully improve uncertainty calibration and enable CoT bypass with minimal performance loss, highlighting latent signals in internal states that can be exploited for efficiency and reliability. The findings suggest practical approaches to uncertainty estimation and conditional reasoning that leverage CoT dynamics without sacrificing accuracy, with potential applications in faster inference and safer model deployment. The work also provides a unified perspective that reconciles prior observations about internal planning and explicit CoT, offering a path for future research into latent signals and their uses in LLMs.

Abstract

This work stems from prior complementary observations on the dynamics of Chain-of-Thought (CoT): Large Language Models (LLMs) is shown latent planning of subsequent reasoning prior to CoT emergence, thereby diminishing the significance of explicit CoT; whereas CoT remains critical for tasks requiring multi-step reasoning. To deepen the understanding between LLM's internal states and its verbalized reasoning trajectories, we investigate the latent planning strength of LLMs, through our probing method, Tele-Lens, applying to hidden states across diverse task domains. Our empirical results indicate that LLMs exhibit a myopic horizon, primarily conducting incremental transitions without precise global planning. Leveraging this characteristic, we propose a hypothesis on enhancing uncertainty estimation of CoT, which we validate that a small subset of CoT positions can effectively represent the uncertainty of the entire path. We further underscore the significance of exploiting CoT dynamics, and demonstrate that automatic recognition of CoT bypass can be achieved without performance degradation. Our code, data and models are released at https://github.com/lxucs/tele-lens.

Figures (24)

• Figure 1: Results for the final-answer probing: average accuracy of In-Domain LLM for the first five tokens within CoT trajectories, measured across selected Transformers layers and tasks. The full figure across all tasks is presented in \ref{['fig:probing-answer-by-layer-full']} (see Appendix \ref{['app:probe-results']}).
• Figure 2: Examples of final-answer probing accuracy along CoT trajectories with In-Domain LLM (random guessing is at 50%). The vertical dashed line indicates the position at which accuracy first spikes. "LEFT" and "RIGHT" at the bottom illustrate the reasoning details right before and after the accuracy spike, respectively. Similar examples with Off-the-Shelf LLM are provided in \ref{['fig:full-32b']}.
• Figure 3: Average final-answer probing accuracy on CSQA with Off-the-Shelf LLM (Qwen3-32B) along CoT positions. The most frequent token at each position is annotated with its occurrence frequency. The notably earlier accuracy spikes are especially pronounced for Knowledge and Semantic tasks, but largely remain flat for Compositional tasks. The full results across all tasks are shown in \ref{['fig:spike-32b-full-a']} for Off-the-Shelf LLM and \ref{['fig:spike-32b-full-b']} for In-Domain LLM (Appendix \ref{['app:probe-results']}) .
• Figure 4: Task accuracy comparison for Off-the-Shelf LLM (Qwen3-32B) under four settings: using thinking mode (w/ CoT); using non-thinking mode (w/o CoT); the best probing accuracy among initial CoT positions (Probing); the random-guess baseline (Random). The coarse signals of early final-answer planning are shown inferior to the direct prediction counterpart without CoT involved. Full results across all tasks are provided in \ref{['fig:acc-32b-full']}. Similar comparisons for In-Domain LLM is provided in \ref{['fig:acc-full']}.
• Figure 5: Top-5 accuracy for subsequent token prediction, using the last Transformers layer of In-Domain LLM. Full results across layers and tasks are presented in \ref{['fig:hop-full']} and \ref{['fig:hop-full-32b']}.