Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models

TL;DR

This paper reframes chain-of-thought reasoning as a dynamic search for the optimal thinking path in large language models. It introduces Neural Chain-of-Thought Search (NCoTS), which uses a dual-factor heuristic to balance correctness and efficiency by selecting thinking operators at decision points, enabling active pruning of suboptimal reasoning branches. Empirical results across diverse benchmarks show Pareto improvements: average accuracy gains exceeding 3.5% and generation length reductions over 22%, with negligible inference overhead due to sparse activation. The work highlights the importance of high-level path planning in reasoning and demonstrates significant practical impact for more reliable and concise LLM reasoning at inference time.

Abstract

Chain-of-Thought reasoning has significantly enhanced the problem-solving capabilities of Large Language Models. Unfortunately, current models generate reasoning steps sequentially without foresight, often becoming trapped in suboptimal reasoning paths with redundant steps. In contrast, we introduce Neural Chain-of-Thought Search (NCoTS), a framework that reformulates reasoning as a dynamic search for the optimal thinking strategy. By quantitatively characterizing the solution space, we reveal the existence of sparse superior reasoning paths that are simultaneously more accurate and concise than standard outputs. Our method actively navigates towards these paths by evaluating candidate reasoning operators using a dual-factor heuristic that optimizes for both correctness and computational cost. Consequently, NCoTS achieves a Pareto improvement across diverse reasoning benchmarks, boosting accuracy by over 3.5% while reducing generation length by over 22%. Our code and data are available at https://github.com/MilkThink-Lab/Neural-CoT-Search.

Figures (11)

• Figure 1: Motivation and Overview of our NCoTS. (a) Planning Bottleneck in Traditional CoT. (b) Importance of Path Planning. Sparse guiding tokens from a strong teacher significantly boost performance, confirming that path planning is the key bottleneck. (c) The NCoTS framework. Our method reformulates reasoning as a search process, employing a dual-factor heuristic to actively discover paths that are both accurate and concise.
• Figure 2: Overview of the Neural Chain-of-Thought Search (NCoTS) Framework. (a) The Path Potential Estimator employs policy distillation from a teacher model to capture high level planning capabilities. (b) The Reasoning Progress Estimator learns to predict the normalized solution progress via token level dense supervision. (c) The search algorithm during inference. The model pauses at decision points to search how to think. It performs a one step lookahead and evaluates candidate thinking tokens using a dual-factor heuristic function.
• Figure 3: Visualization of the reasoning solution space. The region to the upper-left of the Original result indicates the existence of superior solutions. This confirms that paths with higher accuracy and lower length are attainable, validating the feasibility of our search framework. The red cross mark represents our method, demonstrating that our strategy successfully discovers these superior paths that optimize both accuracy and conciseness.
• Figure 4: Correlation between thinking tokens and thinking modes. This Sankey diagram illustrates the strong influence of the chosen operator (thinking token) on the functional purpose of the subsequent reasoning step.
• Figure 5: A comparison of the estimated progress against the ground truth progress. The exponentially smoothed estimator output closely aligns with the ground truth progress $y=x/L$.