Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs

Tristan Cinquin, Geoff Pleiss, Agustinus Kristiadi

TL;DR

This work interrogates whether PRM-guided tree search can outpace Best-of-N strategies for mathematical reasoning in LLMs. By formulating reasoning as a tree-search problem and developing an adaptive, Gittins-index–driven approach with a Bayesian logit-Normal surrogate, the authors evaluate multiple search baselines on 23 problems with a Qwen-based LLM/PRM pair. The results show BoN remains competitive or superior in accuracy and is far more cost-efficient, while MCTS and beam search are the strongest among PRM-guided methods; PRMs fail to reliably estimate intermediate state values and degrade with reasoning depth, with limited generalization to out-of-distribution tasks. The findings indicate that, in current form, PRMs are not yet suitable for guiding dynamic mathematical reasoning via tree search, underscoring the need for new reward modeling approaches that better capture stepwise credit assignment and problem structure. The study highlights the practical impact that robust, task-aligned reward signals must have to enable reliable, scalable improvement to mathematical reasoning through search-based methods.

Abstract

While chain-of-thought prompting with Best-of-N (BoN) selection has become popular for mathematical reasoning in large language models (LLMs), its linear structure fails to capture the branching and exploratory nature of complex problem-solving. In this work, we propose an adaptive algorithm to maximize process reward model (PRM) scores over the intractable action space, and investigate whether PRM-guided tree search can improve mathematical reasoning by exploring multiple partial solution paths. Across $23$ diverse mathematical problems using Qwen2.5-Math-7B-Instruct with its associated PRM as a case study, we find that: (1) PRM-guided tree search shows no statistically significant improvements over BoN despite higher costs, (2) Monte Carlo tree search and beam search outperform other PRM-guided tree search methods, (3) PRMs poorly approximate state values and their reliability degrades with reasoning depth, and (4) PRMs generalize poorly out of distribution. This underperformance stems from tree search's greater reliance on unreliable PRM scores, suggesting different reward modeling is necessary before tree search can effectively enhance mathematical reasoning in LLMs.

Limits of PRM-Guided Tree Search for Mathematical Reasoning with LLMs

TL;DR

This work interrogates whether PRM-guided tree search can outpace Best-of-N strategies for mathematical reasoning in LLMs. By formulating reasoning as a tree-search problem and developing an adaptive, Gittins-index–driven approach with a Bayesian logit-Normal surrogate, the authors evaluate multiple search baselines on 23 problems with a Qwen-based LLM/PRM pair. The results show BoN remains competitive or superior in accuracy and is far more cost-efficient, while MCTS and beam search are the strongest among PRM-guided methods; PRMs fail to reliably estimate intermediate state values and degrade with reasoning depth, with limited generalization to out-of-distribution tasks. The findings indicate that, in current form, PRMs are not yet suitable for guiding dynamic mathematical reasoning via tree search, underscoring the need for new reward modeling approaches that better capture stepwise credit assignment and problem structure. The study highlights the practical impact that robust, task-aligned reward signals must have to enable reliable, scalable improvement to mathematical reasoning through search-based methods.

Abstract

While chain-of-thought prompting with Best-of-N (BoN) selection has become popular for mathematical reasoning in large language models (LLMs), its linear structure fails to capture the branching and exploratory nature of complex problem-solving. In this work, we propose an adaptive algorithm to maximize process reward model (PRM) scores over the intractable action space, and investigate whether PRM-guided tree search can improve mathematical reasoning by exploring multiple partial solution paths. Across diverse mathematical problems using Qwen2.5-Math-7B-Instruct with its associated PRM as a case study, we find that: (1) PRM-guided tree search shows no statistically significant improvements over BoN despite higher costs, (2) Monte Carlo tree search and beam search outperform other PRM-guided tree search methods, (3) PRMs poorly approximate state values and their reliability degrades with reasoning depth, and (4) PRMs generalize poorly out of distribution. This underperformance stems from tree search's greater reliance on unreliable PRM scores, suggesting different reward modeling is necessary before tree search can effectively enhance mathematical reasoning in LLMs.
Paper Structure (32 sections, 12 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 32 sections, 12 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Limitations.
  3. Background
  4. Mathematical reasoning as tree search
  5. Tree search baselines
  6. Methods
  7. Results
  8. Experimental setup
  9. LLM & PRM.
  10. Problems & metrics.
  11. Tree search methods.
  12. Findings
  13. 1. PRM-guided tree search methods do not outperform Best-of-N despite higher costs.
  14. 2. MCTS and beam search perform best among PRM-guided tree search methods.
  15. 3. The PRM poorly approximates ${ p \probparens[]{v(s) = 1 \nonscript\:\delimsize\vert \nonscript\: \mathopen{} s} }$ and reliability degrades with reasoning depth, limiting tree search effectiveness.
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Correlation of prediction correctness with PRM scores. Correlation decreases with increasing distance in reasoning steps from terminal states.
  • Figure 2: Point-biserial correlation between PRM score aggregation methods and solution correctness. Minimum aggregation shows highest correlation, followed by last reasoning-step scoring. Correlation is consistently higher on in-distribution (ID) problems on which the PRM is trained (GSM8K and MATH) than out-of-distribution (OOD) tasks.
  • Figure 3: PRM score distributions by aggregation method, conditioned on prediction correctness. Last, average, minimum, and product aggregations effectively separate correct from incorrect predictions.