Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training

Qi Cao, Pengtao Xie

TL;DR

DreamPRM-1.5 extends domain-level reweighting to instance-level reweighting for multimodal process reward model training, addressing both data quality imbalance and training-test distribution shift. It introduces two regimes, Instance Table and Instance Net, within a stable bi-level optimization framework that optimizes PRM parameters on weighted data while learning weights on a meta set that mimics test-time scaling. The approach achieves state-of-the-art results on MMMU and strong performance on R-Bench-V, with results generalizing across multiple multimodal reasoning benchmarks and backbones. A key finding is that effective instance weighting requires careful time-scale matching and initialization; with sufficient data, Instance Net demonstrates robust, scalable weight adaptation and substantial gains. Overall, DreamPRM-1.5 significantly narrows the gap to oracle performance while maintaining stable training and strong cross-domain performance, offering practical improvements for multimodal PRMs in real-world settings.

Abstract

Training multimodal process reward models (PRMs) is hard due to (i) distribution shift between training set and test set and (ii) quality imbalance across training data samples. While domain-level reweighting (e.g., DreamPRM) aligns training with test-time objectives, it leaves a clear gap to an oracle upper bound (pass@N), even under a "sanity check" that uses test set data to probe headroom -- pointing to meta-level under-parameterization. We introduce DreamPRM-1.5, an instance-level reweighting framework that assigns an adaptive weight to every training example via bi-level optimization. To realize instance reweighting across scales, we develop two complementary regimes: Instance Table, which learns explicit per-sample weights and excels on small/medium data, and Instance Net, a lightweight neural network that generalizes better and scales to large corpora. A practical, stable training recipe -- time-scale matching between upper/lower updates, cold-start initialization, and bounded-range weights -- prevents divergence. Integrated with test-time scaling, DreamPRM-1.5 attains 84.6 accuracy on the MMMU validation set, 31.3 accuracy on R-Bench-V and, when paired with a leading backbone (e.g., GPT-5-mini), achieves first-place results on public multimodal reasoning leaderboards. Moreover, extensive experiments, including benchmark evaluations, baseline comparisons, and a sanity check, demonstrate that DreamPRM-1.5 closes the gap toward the oracle, achieves leading performance, and trains stably.

DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training

TL;DR

DreamPRM-1.5 extends domain-level reweighting to instance-level reweighting for multimodal process reward model training, addressing both data quality imbalance and training-test distribution shift. It introduces two regimes, Instance Table and Instance Net, within a stable bi-level optimization framework that optimizes PRM parameters on weighted data while learning weights on a meta set that mimics test-time scaling. The approach achieves state-of-the-art results on MMMU and strong performance on R-Bench-V, with results generalizing across multiple multimodal reasoning benchmarks and backbones. A key finding is that effective instance weighting requires careful time-scale matching and initialization; with sufficient data, Instance Net demonstrates robust, scalable weight adaptation and substantial gains. Overall, DreamPRM-1.5 significantly narrows the gap to oracle performance while maintaining stable training and strong cross-domain performance, offering practical improvements for multimodal PRMs in real-world settings.

Abstract

Training multimodal process reward models (PRMs) is hard due to (i) distribution shift between training set and test set and (ii) quality imbalance across training data samples. While domain-level reweighting (e.g., DreamPRM) aligns training with test-time objectives, it leaves a clear gap to an oracle upper bound (pass@N), even under a "sanity check" that uses test set data to probe headroom -- pointing to meta-level under-parameterization. We introduce DreamPRM-1.5, an instance-level reweighting framework that assigns an adaptive weight to every training example via bi-level optimization. To realize instance reweighting across scales, we develop two complementary regimes: Instance Table, which learns explicit per-sample weights and excels on small/medium data, and Instance Net, a lightweight neural network that generalizes better and scales to large corpora. A practical, stable training recipe -- time-scale matching between upper/lower updates, cold-start initialization, and bounded-range weights -- prevents divergence. Integrated with test-time scaling, DreamPRM-1.5 attains 84.6 accuracy on the MMMU validation set, 31.3 accuracy on R-Bench-V and, when paired with a leading backbone (e.g., GPT-5-mini), achieves first-place results on public multimodal reasoning leaderboards. Moreover, extensive experiments, including benchmark evaluations, baseline comparisons, and a sanity check, demonstrate that DreamPRM-1.5 closes the gap toward the oracle, achieves leading performance, and trains stably.

Paper Structure

This paper contains 61 sections, 10 equations, 24 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The Proposed Instance-reweighting Method
  3. Bi-level Optimization for Instance Reweighting
  4. Lower-level optimization.
  5. Upper-level optimization.
  6. Discussion.
  7. Model Design for Instance Reweighting
  8. Instance Table.
  9. Instance Net.
  10. Technical details
  11. Generative reward model.
  12. Cold-start initialization.
  13. Multi-turn fine-tuning.
  14. Aggregation function loss.
  15. Experiments
  16. ...and 46 more sections

Figures (24)

  • Figure 1: Instance reweighting approaches the oracle.Left: Gain over pass@1 (right bar) and gap to pass@8 (left bar); x-axis is symmetric in percentage points (smaller gap = closer to oracle). “Sanity check’’ uses the test set as the meta set to probe headroom. Right: Concept: domain reweighting assigns one weight per dataset, whereas instance reweighting (Instance Table/Net) weights each sample, yielding larger gains and a smaller oracle gap (gray arrow → oracle).
  • Figure 2: Comparison of two model designs for instance reweighting in DreamPRM-1.5. Instance Table assigns an explicit learnable weight to each training sample, offering strong per-instance flexibility but scaling with dataset size. Instance Net parameterizes instance weights via a lightweight MLP appended to the PRM, maintaining a fixed parameter size independent of dataset scale and providing better generalization.
  • Figure 3: Leaderboard on MMMU as of September 16, 2025. Results are taken from the official MMMU leaderboard validation set yue2024mmmumassivemultidisciplinemultimodal, while the scores of DreamPRM-1.5 (Instance Table and Instance Net) and the base GPT-5-mini are measured by us.
  • Figure 4: Leaderboard on R-Bench-V as of October 20, 2025. Results are taken from the official R-Bench-V guo2025rbenchvprimaryassessmentvisual, while the scores of DreamPRM-1.5 (Instance Table and Instance Net) and the base GPT-5-mini are measured by us.
  • Figure 5: Training loss of instance table. The figure shows the convergence of the training loss, indicating stable optimization and steady improvement over time.
  • ...and 19 more figures