What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning

TL;DR

<3-5 sentence high-level summary> The paper investigates step-level reward models (SRMs) for mathematical reasoning using MCTS-derived step preferences, formalizing a math-reasoning MDP and several SRM variants. It demonstrates that natural language descriptions of thought processes are largely unnecessary for effective SRMs, and that SRMs trained on mathematical language can assess logical coherence within math expressions, sometimes outperforming NL-inclusive variants. The work shows MO-SRM often matches or surpasses full-context SRMs, and that larger base models can further boost performance, while NL coherence remains challenging. These findings point to more efficient SRM design that focuses on mathematical structure, enabling faster, more scalable guidance for LLM-based math reasoning and beam-search-assisted problem solving.

Abstract

Step-level reward models (SRMs) can significantly enhance mathematical reasoning performance through process supervision or step-level preference alignment based on reinforcement learning. The performance of SRMs is pivotal, as they serve as critical guidelines, ensuring that each step in the reasoning process is aligned with desired outcomes. Recently, AlphaZero-like methods, where Monte Carlo Tree Search (MCTS) is employed for automatic step-level preference annotation, have proven particularly effective. However, the precise mechanisms behind the success of SRMs remain largely unexplored. To address this gap, this study delves into the counterintuitive aspects of SRMs, particularly focusing on MCTS-based approaches. Our findings reveal that the removal of natural language descriptions of thought processes has minimal impact on the efficacy of SRMs. Furthermore, we demonstrate that SRMs are adept at assessing the complex logical coherence present in mathematical language while having difficulty in natural language. These insights provide a nuanced understanding of the core elements that drive effective step-level reward modeling in mathematical reasoning. By shedding light on these mechanisms, this study offers valuable guidance for developing more efficient and streamlined SRMs, which can be achieved by focusing on the crucial parts of mathematical reasoning.

Figures (7)

• Figure 1: Each step in an LLM’s process of solving mathematical problems can be divided into the thought process and the execution of corresponding calculations. We find that natural language descriptions of the thought processes are not essential for step-level reward modeling.
• Figure 2: Illustration of the role of SRMs in mathematical reasoning and the SRMs with different input structures we investigate.
• Figure 3: Effect of natural language descriptions and math expressions on step-level reward modeling. The agent and the environment model is Llama-3-8B-Instruct. The reward models are trained based on Qwen2-7B or Deepseek-Math-7B-Base. (Note that the 'accuracy' here is the accuracy of preference during reward training.)
• Figure 4: SRMs take only mathematical expressions as input demonstrate the same ability during the greedy search as those take full context as input. The boxplot is obtained through 20 runs over the dataset.
• Figure 5: The performance of SRM is affected by the ability of the base model.