Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

TL;DR

The paper proposes UnifiedReward-Think, a unified multimodal reward model that injects explicit long-chain thought processes into reward reasoning to improve reliability for vision understanding and generation tasks. It introduces a three-stage exploration-driven training pipeline—cold start with CoT distillation from GPT-4o, rejection sampling for generalization, and GRPO-based reinforcement fine-tuning with verifiable rewards—to cultivate multi-dimensional CoT reasoning. A multidimensional reward scoring strategy aligns reasoning with final decisions, addressing inconsistencies between CoT traces and outcomes. Experimental results show significant gains over baselines across image/video generation and understanding, with the model also exhibiting strong implicit reasoning even when CoT traces are not produced. The work advances interpretability, generalization, and alignment of multimodal reward signals, enabling more trustworthy and human-aligned guidance for vision-based systems.

Abstract

Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments across various vision reward tasks demonstrate the superiority of our model.

Figures (8)

• Figure 1: Overview of Comparison Results. (a) Our method enables multi-dimensional long CoT reasoning to improve reward signal accuracy. (b) Extensive quantitative results demonstrate our superiority in both vision understanding and generation reward tasks.
• Figure 2: Method Overview. (1) Cold Start: We first distill GPT-4o's reasoning process on a small amount of labeled image generation preference data to initialize the model's CoT reasoning format; (2) Rejection Sampling: Then, we leverage the model's generalization capabilities on large-scale unified multimodal preference data to elicit its CoT reasoning process across various vision tasks. Correctly predicted samples whose final answers match the ground-truth are retained for rejection sampling to further refine the model. (3) GRPO: Finally, incorrectly predicted samples are utilized for GRPO-based reinforcement fine-tuning to further enhance step-by-step reasoning capabilities.
• Figure 3: Qualitative Results of Video Generation CoT Reward Reasoning. Given a pair of videos and the corresponding caption, our model performs quality assessment across semantic consistency, temporal coherence, and authenticity through CoT reasoning.
• Figure 4: Qualitative Cases of Image and Video Understanding CoT Reward Reasoning. Given an image or video, a query, and a pair of candidate answers, our model performs quality assessment across semantic accuracy, factual correctness, and clarity through CoT reasoning.
• Figure 5: More Qualitative Results of Image Generation CoT Reward Reasoning. Given a pair of images and the corresponding caption, our model performs quality assessment across semantic consistency, aesthetics, and authenticity through CoT reasoning.