Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search

Pengfei Hu, Zhenrong Zhang, Qikai Chang, Shuhang Liu, Jiefeng Ma, Jun Du, Jianshu Zhang, Quan Liu, Jianqing Gao, Feng Ma, Qingfeng Liu

TL;DR

PRM-BAS tackles the challenge of guiding multimodal reasoning in LLMs by introducing a PRM-guided beam annealing search that dynamically adjusts the search width as reasoning progresses. The authors construct PRM-BAS-300k through automated, rollout-based data sampling and train a PRM with both value and rank losses to predict step-level rewards, enabling more reliable stepwise evaluation. Empirical results across MathVista, MathVision, ChartQA, and M3CoT demonstrate significant reasoning accuracy gains with controllable computational cost, and the approach generalizes across model scales though shows a policy-alignment dependence. The work provides a practical, plug-and-play framework for data construction and PRM learning, highlighting the importance of soft, process-level supervision and early-stage exploration efficiency for robust multimodal reasoning.

Abstract

Recent work increasingly focuses on improving the reasoning capabilities of Multimodal Large Language Models (MLLMs). Among existing methods, Process Reward Models (PRMs) stand out for offering dense, step-wise supervision to guide intermediate reasoning. However, how to effectively integrate PRMs into search strategies remains an open question. In this paper, we introduce PRM-BAS (PRM-Guided Beam Annealing Search), a lightweight approach for PRM-guided reasoning that dynamically adjusts beam size -- starting with a broader search space and gradually narrowing it as contextual information accumulates, thereby balancing performance and efficiency. We further propose a unified framework for data construction and PRM training. Specifically, we construct the PRM-BAS-300k dataset by selecting 300k questions from existing datasets and performing rollouts at each step to estimate the probability of reaching a correct final answer. The PRM is then trained using a combination of value loss for absolute action quality and rank loss for relative action quality. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate that PRM-BAS significantly improves reasoning performance while maintaining low computational cost. Moreover, it generalizes well across different model scales and architectures, showcasing strong robustness and plug-and-play capability.

PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search

TL;DR

PRM-BAS tackles the challenge of guiding multimodal reasoning in LLMs by introducing a PRM-guided beam annealing search that dynamically adjusts the search width as reasoning progresses. The authors construct PRM-BAS-300k through automated, rollout-based data sampling and train a PRM with both value and rank losses to predict step-level rewards, enabling more reliable stepwise evaluation. Empirical results across MathVista, MathVision, ChartQA, and M3CoT demonstrate significant reasoning accuracy gains with controllable computational cost, and the approach generalizes across model scales though shows a policy-alignment dependence. The work provides a practical, plug-and-play framework for data construction and PRM learning, highlighting the importance of soft, process-level supervision and early-stage exploration efficiency for robust multimodal reasoning.

Abstract

Recent work increasingly focuses on improving the reasoning capabilities of Multimodal Large Language Models (MLLMs). Among existing methods, Process Reward Models (PRMs) stand out for offering dense, step-wise supervision to guide intermediate reasoning. However, how to effectively integrate PRMs into search strategies remains an open question. In this paper, we introduce PRM-BAS (PRM-Guided Beam Annealing Search), a lightweight approach for PRM-guided reasoning that dynamically adjusts beam size -- starting with a broader search space and gradually narrowing it as contextual information accumulates, thereby balancing performance and efficiency. We further propose a unified framework for data construction and PRM training. Specifically, we construct the PRM-BAS-300k dataset by selecting 300k questions from existing datasets and performing rollouts at each step to estimate the probability of reaching a correct final answer. The PRM is then trained using a combination of value loss for absolute action quality and rank loss for relative action quality. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate that PRM-BAS significantly improves reasoning performance while maintaining low computational cost. Moreover, it generalizes well across different model scales and architectures, showcasing strong robustness and plug-and-play capability.

Paper Structure

This paper contains 19 sections, 5 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt-based Methods
  4. Learning-based Methods
  5. Search-based Methods
  6. Method
  7. Problem Formulation
  8. Data Construction
  9. PRM Training
  10. PRM-guided Search
  11. Experiments
  12. Datasets
  13. Implementation Details
  14. Comparison With State-of-the-Art Methods
  15. Ablation Study
  16. ...and 4 more sections

Figures (4)

  • Figure 1: (a) Average accuracy across common benchmarks. (b) Test-Time Scaling curves under different token consumption ratios relative to single-shot inference.
  • Figure 2: (a) The illustration of data construction. (b) The illustration of PRM training.
  • Figure 3: (a) The statistics of step-wise reward during data construction, including the mean and variance of $r_t$, and the training loss of PRM across reasoning steps. (b) The comparison of different PRM-guided search strategies.
  • Figure 4: Qualitative case study on a geometry problem. PRM-BAS identifies and rejects steps with hallucinations and reasoning errors, guiding the policy model toward the correct conclusion.