Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Adaptive Reasoning in Large Language Models with Thought Rollback

Sijia Chen, Baochun Li

TL;DR

This paper introduces Thought Rollback (TR), a framework that enables adaptive reasoning in large language models by allowing thoughts to roll back to earlier steps for error analysis and revision. A rollback controller identifies bad intermediate thoughts using a dedicated reasoning-analysis prompt, while a prompt enhancer accumulates these error analyses as experience to guide future thoughts, forming directed graphs with cycles rather than fixed chains or trees. TR supports multiple reasoning paths and ensembles final solutions via weighted voting, achieving state-of-the-art or competitive results on math and multi-task benchmarks with significantly reduced interaction costs compared to some tree-based methods. The approach emphasizes process-level supervision over outcome-level feedback, accepting higher token costs in exchange for substantially improved robustness against hallucinations and better problem-solving performance in challenging tasks.

Abstract

Large language models (LLMs) have been routinely used to solve various tasks using step-by-step reasoning. However, the structure of intermediate reasoning steps, or thoughts, is rigid and unidirectional, such as chains, trees, or acyclic-directed graphs. Consequently, the resulting inflexible and forward-only reasoning may not address challenging tasks and fail when the LLM frequently gives false responses, i.e., ``hallucinations''. This paper proposes a new reasoning framework, called Thought Rollback (TR), allowing LLMs to adaptively build thought structure while maintaining effective reasoning toward problem-solving under ``hallucinations''. The core mechanism of TR is rolling back thoughts, which allows LLMs to perform error analysis on thoughts, and thus roll back to any previously mistaken thought for revision. Subsequently, by including such trial-and-error in the prompt to guide the LLM, each rollback leads to one more reliable reasoning path. Therefore, starting with a simple prompt without human annotations, LLM with TR adaptively and gradually explores thoughts for a correct solution. Comprehensive experiments on mathematical problems and multi-task reasoning demonstrate the state-of-the-art performance of TR in terms of problem-solving rate and interaction cost. For instance, the solving rate of GPT-4 with TR outperforms the current best by $9\%$ on the MATH dataset.

Toward Adaptive Reasoning in Large Language Models with Thought Rollback

TL;DR

This paper introduces Thought Rollback (TR), a framework that enables adaptive reasoning in large language models by allowing thoughts to roll back to earlier steps for error analysis and revision. A rollback controller identifies bad intermediate thoughts using a dedicated reasoning-analysis prompt, while a prompt enhancer accumulates these error analyses as experience to guide future thoughts, forming directed graphs with cycles rather than fixed chains or trees. TR supports multiple reasoning paths and ensembles final solutions via weighted voting, achieving state-of-the-art or competitive results on math and multi-task benchmarks with significantly reduced interaction costs compared to some tree-based methods. The approach emphasizes process-level supervision over outcome-level feedback, accepting higher token costs in exchange for substantially improved robustness against hallucinations and better problem-solving performance in challenging tasks.

Abstract

Large language models (LLMs) have been routinely used to solve various tasks using step-by-step reasoning. However, the structure of intermediate reasoning steps, or thoughts, is rigid and unidirectional, such as chains, trees, or acyclic-directed graphs. Consequently, the resulting inflexible and forward-only reasoning may not address challenging tasks and fail when the LLM frequently gives false responses, i.e., ``hallucinations''. This paper proposes a new reasoning framework, called Thought Rollback (TR), allowing LLMs to adaptively build thought structure while maintaining effective reasoning toward problem-solving under ``hallucinations''. The core mechanism of TR is rolling back thoughts, which allows LLMs to perform error analysis on thoughts, and thus roll back to any previously mistaken thought for revision. Subsequently, by including such trial-and-error in the prompt to guide the LLM, each rollback leads to one more reliable reasoning path. Therefore, starting with a simple prompt without human annotations, LLM with TR adaptively and gradually explores thoughts for a correct solution. Comprehensive experiments on mathematical problems and multi-task reasoning demonstrate the state-of-the-art performance of TR in terms of problem-solving rate and interaction cost. For instance, the solving rate of GPT-4 with TR outperforms the current best by on the MATH dataset.
Paper Structure (27 sections, 12 figures, 6 tables)

This paper contains 27 sections, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Problem Statement
  5. Motivation: Forward-only reasoning fails in bad thoughts
  6. Motivation: Error analysis induces better thoughts
  7. Thought Rollback Framework
  8. Reasoning Overview
  9. Rolling Back Thought with Reasoning Analysis
  10. Enhancing the Prompt with Errors as Experience
  11. Ensembling Solutions
  12. Experiments
  13. Main Evaluation Results
  14. Main Insights
  15. Token Cost Analysis
  16. ...and 12 more sections

Figures (12)

  • Figure 1: Schematic illustrating thought structures for problem solving with GPT-4. The chain, tree, and our thought rollback (TR) structures are plotted based on the NetworkX lib netx-08. The question from the MATH dataset MATH-arxiv21 is: I draw a card from a standard 52-card deck. If I draw an Ace, I win 1 dollar. If I draw a 2 through 10, I win a number of dollars equal to the value of the card. If I draw a face card (Jack, Queen, or King), I win 20 dollars. If I draw a 'clubsuit', my winnings are doubled, and if I draw a 'spadesuit', my winnings are tripled. ... What would be a fair price to pay to play the game? In (c), we present a partial thought structure built by GPT-4 with TR and place the full version of the TR structure in Figure \ref{['fig:introcomplete']} of the Appendix.
  • Figure 2: Schematic illustrating the rollback of thought when the LLM with TR reaches the $n$-th reasoning step. We add $A^{\chi\left(\cdot\right)}_{z_{0...m-1}}$ in the reasoning from $z_{m-1}$ to $z_m$ to cover the case that $z_{m-1}$ may also derive from a rollback. We present a clear example from SVAMP in Figure \ref{['fig:fig2example']} of the Appendix.
  • Figure 3: Effectiveness of TR on interaction cost saving and thought revision through the rolling back of thoughts. a). distributions of the interactions required to address problems in four datasets; b). solving rates of three types of reasoning path: "No Rollback" --- thoughts receive no rollbacks, "Out Rollback" --- rollbacks triggered by mistaken thoughts, and "In Rollback" --- thoughts derive from rollbacks; c). the reduction of failure rates due to the rollbacks in the first step of Game of 24, where $\left[0, 1, 3, 5, 7\right]$ denotes the number of rollbacks that cause the first step and $2\rightarrow 0$ means the first step derives from the rollback from the $2$-th step.
  • Figure 4: Complete thought structure of Figure \ref{['fig:intro']} (c) built by GPT-4 with TR for the question from MATH dataset MATH-arxiv21. This structure contains $23$ nodes, i.e. $23$ thoughts and leads to $K=8$ reasoning paths towards solutions. It is plotted by based on the NetworkX lib netx-08 under the "fdp" layout.
  • Figure 5: Complete thought structure of Figure \ref{['fig:tr']} built by GPT-4 with TR for the question from SVAMP dataset SVAMP-arxiv21. This structure contains $10$ nodes, i.e. $10$ thoughts and leads to $K=3$ reasoning paths towards solutions. It is plotted by based on the NetworkX lib netx-08 under the "dot" layout.
  • ...and 7 more figures