Policy of Thoughts: Scaling LLM Reasoning via Test-time Policy Evolution

TL;DR

Policy of Thoughts (PoT) introduces test-time policy evolution to overcome the frozen-policy limitation in LLMs for long-horizon reasoning. By pairing structured exploration (MCTS) with policy internalization through Group Relative Policy Optimization (GRPO) on transient LoRA adapters, PoT evolving the reasoning prior on a per-instance basis. Empirically, a 4B model with PoT achieves substantial gains on code reasoning benchmarks, notably 58.98% average accuracy under matched budgets and surpassing several frontier models. The approach demonstrates strong cross-architecture generalization and broad applicability to environment-grounded and RM-assisted tasks, highlighting test-time adaptation as a powerful, scalable paradigm for reasoning beyond static policies.

Abstract

Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.

Figures (4)

• Figure 1: The Policy of Thoughts (PoT) Framework: A Reasoning System inspired by Popperian epistemology. (a) The reasoning cycle:P1 (identify problem), TT (propose conjecture), EE (test against reality), and P2 (update understanding). (b) PoT implementation: The model generates solutions (TT); the environment evaluates (EE); feedback is internalized via RL to update the reasoning policy (P2)—enabling real-time adaptation.
• Figure 2: Comparison of model scale versus Live evaluation accuracy. Our PoT-enhanced model achieves 49.71% accuracy, significantly outperforming larger parameter scales—where accuracy degrades with increasing parameters.
• Figure 3: Policy evolution on a tree with frozen LLM and dynamic LoRA adaptation. (a) Overall pipeline: A thought tree is constructed rooted at the problem. The LLM backbone remains frozen throughout, while the policy is represented and evolved through LoRA adapters. In each iteration, multiple children are expanded from pending nodes, evaluated by the environment, and low-quality branches are pruned to guide subsequent exploration. This process repeats until convergence to a solution. (b) Internal mechanism of a single iteration: Corresponding to any iteration in (a), all child nodes sharing the same parent form a group. Their trajectories are generated in parallel, evaluated by the environment, and used to compute group-wise advantage signals. The LoRA parameters are then updated via GRPO based on this intra-group comparison, enabling structured policy refinement.
• Figure 4: Overall performance on reasoning benchmarks. PoT achieves the highest score (58.98%), significantly outperforming self-refinement methods, search-based approaches, and standard inference baselines. It also surpasses recent MCTS-enhanced methods such as AB-MCTS and ReinforceMCTS, demonstrating the effectiveness of its unified thought-tree-guided policy evolution framework.