Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned

TL;DR

The paper addresses the reliability of multimodal reasoning in Vision-Language Models by extending Process Reward Models (PRMs) to the vision-language domain and optimizing test-time scaling (TTS). It introduces VL-PRM300K, a structured, perception-aware VL-PRM dataset built with MCTS and an external judge, and trains PRMs to evaluate each reasoning step. The study compares three TTS strategies, demonstrates that perception-focused supervision boosts performance, and shows VL-PRMs reveal latent reasoning abilities in stronger backbones across five multimodal benchmarks. It also provides PerceptionProcessBench to specifically test perception error detection and conducts extensive ablations showing the superiority of PRMs over direct LLM judges and greedy decoding in many settings. Overall, the work offers practical insights into VL-PRM design, training, and inference that improve accuracy and generalization in multimodal reasoning tasks.

Abstract

Process Reward Models (PRMs) provide step-level supervision that improves the reliability of reasoning in large language models. While PRMs have been extensively studied in text-based domains, their extension to Vision Language Models (VLMs) remains limited. Existing Vision-Language PRMs (VL-PRMs) rely on Monte Carlo Tree Search (MCTS) for data construction, which can often produce noisy supervision signals and limit generalization across tasks. In this work, we aim to elucidate the design space of VL-PRMs by exploring diverse strategies for dataset construction, training, and test-time scaling. First, we introduce a hybrid data synthesis framework that combines MCTS with judgments from a strong VLM, producing more accurate step-level labels. Second, we propose perception-focused supervision, enabling our PRM to explicitly detect errors at the visual grounding stage of reasoning. Third, we systematically evaluate multiple test-time scaling strategies, showing that our PRMs can reliably guide VLMs toward more accurate solutions. Our experiments covering five diverse multimodal benchmarks (MMMU, PuzzleVQA, AlgoPuzzleVQA, MathVista, and MathVision) reveal several key insights: (i) VL-PRMs when used as Outcome Reward Models (ORMs) during test-time scaling (TTS) can outperform VL-PRM guided process step selection, (ii) smaller VL-PRMs can match or even surpass larger ones in detecting process errors, (iii) VL-PRMs uncover latent reasoning abilities in stronger VLM backbones, (iv) perception-level supervision leads to significant gains in test-time scaling, and (v) TTS performance of different policies improve on advanced math reasoning datasets despite not training VL-PRMs on such datasets. We hope our work will motivate further research and support the advancement of VLMs.

Figures (11)

• Figure 1: Visual and reasoning errors detected by MC scoring and o4-mini judge in VL-PRM300K.
• Figure 2: Comparison among different TTS methods.
• Figure 3: MC-Only vs o4-mini judge for VL-PRM300K construction.
• Figure 4: Different TTS strategies.
• Figure 5: Comparison between BON@8 and BON@16 across datasets and models.