Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought

TL;DR

The paper tackles the problem of noisy feedback in chain-of-thought generation by introducing a pairwise-comparison framework (C-ToT) that iteratively identifies the most promising intermediate thoughts. It retains previous thoughts to mitigate noise and presents two instantiations: Standard (direct pairwise voting) and Duel (dueling bandits) to handle varying noise levels. The approach is supported by theoretical analysis of comparison efficiency and PAC-like guarantees, and validated across QA, the Game of 24, and Sudoku, where C-ToT variants outperform traditional CoT, SC-CoT, and SToT baselines. This yields a more robust, cost-aware method for guiding LLM-based reasoning in complex tasks, with broad applicability to multi-step inference problems.

Abstract

To improve the ability of the large language model (LLMs) to tackle complex reasoning problems, chain-of-thoughts (CoT) methods were proposed to guide LLMs to reason step-by-step, enabling problem solving from simple to complex. State-of-the-art methods for generating such a chain involve interactive collaboration, where the learner generates candidate intermediate thoughts, evaluated by the LLM, guiding the generation of subsequent thoughts. However, a widespread yet understudied problem is that the evaluation from the LLM is typically noisy and unreliable, potentially misleading the generation process in selecting promising intermediate thoughts. In this paper, motivated by Vapnik's principle, we use pairwise-comparison evaluation instead of point-wise scoring to search for promising intermediate thoughts with the noisy feedback from the LLM. In each round, we randomly pair intermediate thoughts and directly prompt the LLM to select the more promising one from each pair, allowing us to identify the most promising thoughts through an iterative process. To further alleviate the noise in the comparison, we incorporate techniques from ensemble learning and dueling bandits, proposing two variants of the algorithm. Experiments on three real-world tasks demonstrate the effectiveness of our proposed algorithm and verify the rationale of the pairwise comparison mechanism.

Figures (15)

• Figure 1: A demonstration of point-wise evaluation vs. pair-wise comparison based on real experimental results in Sudoku puzzles. The point-wise evaluation algorithm (S-ToT) assigns hard scores to each intermediate thought (the higher, the better), while our proposed algorithm (C-ToT) uses pair-wise comparison to obtain the more promising thoughts (green tick). In these cases, the LLM assigns incorrect scores, but it makes a correct comparison.
• Figure 1: Average accuracy on AQuA.
• Figure 2: Schematic illustration of previous CoT and S-ToT approaches with our proposed C-ToT approach for CoT generation with LLMs. Each circle box represents an intermediate thought, which is a coherent sequence of language or equations that serves as an intermediate step in problem solving. In the S-ToT method, each intermediate thought is scored by the LLM (denoted by $s$ in the figure), and the searching algorithm considers the highest-scoring ones as the most promising and then generates next intermediate thoughts based on them. In the C-ToT approach, we use pairwise comparison with the LLM in each round to find the most promising intermediate thoughts and then generate the next thoughts. Meanwhile, we include all previous intermediate thoughts in the comparison.
• Figure 2: Average accuracy on Game of 24.
• Figure 3: Average accuracy on Sudoku Puzzles.

Theorems & Definitions (9)

• Remark 1: Comparison Complexity
• Remark 2: Token Cost
• Lemma 1: Theorem 3 in ICML2017:Knockout
• Proposition 1
• Remark 3
• Lemma 2: Lemma 2 in ICML2017:Knockout
• proof : Proof of Lemma \ref{['lem:knockout_round']}
• proof : Proof of Lemma \ref{['Lemma:Knockout']}
• proof : Proof of Proposition \ref{['Prop:CToT']}