Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solving

Haoyuan Wu, Xueyi Chen, Rui Ming, Jilong Gao, Shoubo Hu, Zhuolun He, Bei Yu

TL;DR

Tree-of-thoughts RL (ToTRL), a novel on-policy RL framework with a rule-based reward, is introduced, designed to guide LLMs in developing the parallel ToT strategy based on the sequential CoT strategy.

Abstract

Large language models (LLMs) demonstrate significant reasoning capabilities, particularly through long chain-of-thought (CoT) processes, which can be elicited by reinforcement learning (RL). However, prolonged CoT reasoning presents limitations, primarily verbose outputs due to excessive introspection. The reasoning process in these LLMs often appears to follow a trial-and-error methodology rather than a systematic, logical deduction. In contrast, tree-of-thoughts (ToT) offers a conceptually more advanced approach by modeling reasoning as an exploration within a tree structure. This reasoning structure facilitates the parallel generation and evaluation of multiple reasoning branches, allowing for the active identification, assessment, and pruning of unproductive paths. This process can potentially lead to improved performance and reduced token costs. Building upon the long CoT capability of LLMs, we introduce tree-of-thoughts RL (ToTRL), a novel on-policy RL framework with a rule-based reward. ToTRL is designed to guide LLMs in developing the parallel ToT strategy based on the sequential CoT strategy. Furthermore, we employ LLMs as players in a puzzle game during the ToTRL training process. Solving puzzle games inherently necessitates exploring interdependent choices and managing multiple constraints, which requires the construction and exploration of a thought tree, providing challenging tasks for cultivating the ToT reasoning capability. Our empirical evaluations demonstrate that our ToTQwen3-8B model, trained with our ToTRL, achieves significant improvement in performance and reasoning efficiency on complex reasoning tasks.

ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solving

TL;DR

Tree-of-thoughts RL (ToTRL), a novel on-policy RL framework with a rule-based reward, is introduced, designed to guide LLMs in developing the parallel ToT strategy based on the sequential CoT strategy.

Abstract

Large language models (LLMs) demonstrate significant reasoning capabilities, particularly through long chain-of-thought (CoT) processes, which can be elicited by reinforcement learning (RL). However, prolonged CoT reasoning presents limitations, primarily verbose outputs due to excessive introspection. The reasoning process in these LLMs often appears to follow a trial-and-error methodology rather than a systematic, logical deduction. In contrast, tree-of-thoughts (ToT) offers a conceptually more advanced approach by modeling reasoning as an exploration within a tree structure. This reasoning structure facilitates the parallel generation and evaluation of multiple reasoning branches, allowing for the active identification, assessment, and pruning of unproductive paths. This process can potentially lead to improved performance and reduced token costs. Building upon the long CoT capability of LLMs, we introduce tree-of-thoughts RL (ToTRL), a novel on-policy RL framework with a rule-based reward. ToTRL is designed to guide LLMs in developing the parallel ToT strategy based on the sequential CoT strategy. Furthermore, we employ LLMs as players in a puzzle game during the ToTRL training process. Solving puzzle games inherently necessitates exploring interdependent choices and managing multiple constraints, which requires the construction and exploration of a thought tree, providing challenging tasks for cultivating the ToT reasoning capability. Our empirical evaluations demonstrate that our ToTQwen3-8B model, trained with our ToTRL, achieves significant improvement in performance and reasoning efficiency on complex reasoning tasks.
Paper Structure (26 sections, 7 equations, 5 figures, 13 tables)

This paper contains 26 sections, 7 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Tree-of-Thoughts RL
  3. On-policy RL Algorithm
  4. Reward Modeling
  5. Multi-Stage ToTRL
  6. LLM as Puzzle Game Player
  7. Experiments
  8. Experiment Settings
  9. In-Distribution Logic Reasoning
  10. Out-of-Distribution Logic Reasoning
  11. Out-of-Distribution Mathematical Tasks
  12. Test Time Scaling
  13. Ablation Studies
  14. Conclusion
  15. Related Works
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Overview of multi-stage ToT guidance with solving an Alphametic puzzle as an example.
  • Figure 2: During the training process with ToTRL, LLMs are employed as players in puzzle games, including Sudoku and Alphametic puzzles.
  • Figure 3: Illustration of test time scaling on logic reasoning tasks.
  • Figure 4: Illustration of CoT reasoning pattern with solving an Alphametic puzzle as an example.
  • Figure 5: Illustration of ToT reasoning pattern with solving an Alphametic puzzle as an example.