R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning

TL;DR

This work reframes reward modeling for multimodal LLMs as a rule-based RL task and introduces StableReinforce to address instability in PPO/Reinforce++ when optimizing MRMs. By collecting 200K preference data and employing a GPT-4o–assisted cold-start plus a consistency/referee mechanism, the authors train R1-Reward to achieve state-of-the-art results on VL Reward-Bench and Multimodal Reward Bench, with notable data efficiency and test-time scalability via voting. The key contributions include Pre-Clip, an Advantage Filter, and a consistency-based reward design that align reasoning with outcomes, enabling robust long-term reasoning in rewards. The results demonstrate the practical viability of RL-based reward modeling for MRMs and point to future improvements in training strategies and inference-time aggregation.

Abstract

Multimodal Reward Models (MRMs) play a crucial role in enhancing the performance of Multimodal Large Language Models (MLLMs). While recent advancements have primarily focused on improving the model structure and training data of MRMs, there has been limited exploration into the effectiveness of long-term reasoning capabilities for reward modeling and how to activate these capabilities in MRMs. In this paper, we explore how Reinforcement Learning (RL) can be used to improve reward modeling. Specifically, we reformulate the reward modeling problem as a rule-based RL task. However, we observe that directly applying existing RL algorithms, such as Reinforce++, to reward modeling often leads to training instability or even collapse due to the inherent limitations of these algorithms. To address this issue, we propose the StableReinforce algorithm, which refines the training loss, advantage estimation strategy, and reward design of existing RL methods. These refinements result in more stable training dynamics and superior performance. To facilitate MRM training, we collect 200K preference data from diverse datasets. Our reward model, R1-Reward, trained using the StableReinforce algorithm on this dataset, significantly improves performance on multimodal reward modeling benchmarks. Compared to previous SOTA models, R1-Reward achieves a $8.4\%$ improvement on the VL Reward-Bench and a $14.3\%$ improvement on the Multimodal Reward Bench. Moreover, with more inference compute, R1-Reward's performance is further enhanced, highlighting the potential of RL algorithms in optimizing MRMs.

Figures (6)

• Figure 1: R1-Reward performance on multimodal reward benchmarks. Performance improves significantly when using a majority voting strategy (Voting@$5/15$) over multiple inference samples.
• Figure 2: Detailed comparison between StableReinforce and Reinforce++. (a) StableReinforce exhibits faster and more stable convergence of the policy loss during training. (b) StableReinforce continuously performs length compression, improving efficiency. Reinforce++ collapses around step 150, whereas StableReinforce remains stable, demonstrating its enhanced robustness. Additionally, after RL training with StableReinforce, the average response length is reduced by approximately 15% compared to base model, suggesting potential improvements in reasoning token efficiency.
• Figure 3: An example of the R1-Reward output. R1-Reward shows "human-like" self-reflective thought process, which is also called "Aha moment" deepseekai2025deepseekr1incentivizingreasoningcapability.
• Figure 4: Inference-time performance scaling of R1-Reward on three benchmarks: (a) MM-RLHF Reward Bench, (b) VL Reward Bench, and (c) Multimodal Reward Bench. Accuracy is measured using two aggregation strategies as the number of inference samples (K) increases: "Majority Vote" and "Any Correct". The "Any Correct" strategy (successful if at least one of the K samples is correct) is highly sensitive to K, while the "Majority Vote" strategy shows a more gradual improvement. Performance is compared against the previous SOTA result for each benchmark.
• Figure 5: Ablation studies of the StableReinforce algorithm, evaluating the impact of different components on policy loss and model response length. The subfigures compare the performance of the algorithm with and without specific components: (a) and (b) show results when the advantage filter is removed; (c) and (d) when Pre Clip is removed. Each method is evaluated in terms of policy loss and response length over training steps.