RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

TL;DR

This work tackles the challenge of improving complex LLM reasoning without modifying model parameters. By framing long-horizon reasoning as an MDP and training a tiny navigator to select and cascade five logic blocks, RLoT constructs task-specific reasoning trajectories at inference time. Empirical results across math, STEM, and commonsense benchmarks show consistent gains over standard inference-time methods, with strong transferability across LLMs and tasks and notable efficiency since the navigator has fewer than $3{,}000$ parameters. The approach offers a practical, scalable path to enhance reasoning in sub-10B LLMs and beyond, and the authors provide open-source code for reproducibility.

Abstract

Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities. To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through sophisticated logical structures without modifying LLMs' parameters. However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability. To improve this, we propose RL-of-Thoughts (RLoT), where we train a lightweight navigator model with reinforcement learning (RL) to adaptively enhance LLM reasoning at inference time. Specifically, we design five basic logic blocks from the perspective of human cognition. During the reasoning process, the trained RL navigator dynamically selects the suitable logic blocks and combines them into task-specific logical structures according to problem characteristics. Experiments across multiple reasoning benchmarks (AIME, MATH, GPQA, etc.) with multiple LLMs (GPT, Llama, Qwen, and DeepSeek) illustrate that RLoT outperforms established inference-time techniques by up to 13.4%. Remarkably, with less than 3K parameters, our RL navigator is able to make sub-10B LLMs comparable to 100B-scale counterparts. Moreover, the RL navigator demonstrates strong transferability: a model trained on one specific LLM-task pair can effectively generalize to unseen LLMs and tasks. Our code is open-source at https://github.com/tsinghua-fib-lab/RL-LLM-Reasoning for reproducibility.

Figures (9)

• Figure 1: Framework of RL-of-Thoughts (RLoT). We train an RL agent as the navigator, which dynamically selects and combines basic logic blocks along the reasoning process, constructing task-specific logical structures for each task and thereby enhancing the LLMs' ability to handle complex reasoning tasks.
• Figure 2: Learning curves during RL training of navigator models.
• Figure 3: Learning curves during RL training of navigator models.
• Figure 4: 5k episodes learning curves during RL training with different algorithms.
• Figure 5: A case study comparing few-shot CoT and RLoT on a representative problem in the MATH dataset. Results are both generated by Qwen2.5-14B-Instruct.