Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning

TL;DR

Skywork-VL Reward introduces a robust multimodal reward model for vision-language understanding and reasoning by building a large, carefully curated preference dataset and employing a two-stage fine-tuning regime on a base VLM. It achieves state-of-the-art results on VL-RewardBench and competitive performance on RewardBench, demonstrating strong generalization across multimodal tasks and text-only scenarios. The authors also show that preference data from Skywork-VL Reward substantially boosts Mixed Preference Optimization, improving multimodal reasoning capabilities. Public release of the model and data promotes transparency and reproducibility in multimodal alignment research.

Abstract

We propose Skywork-VL Reward, a multimodal reward model that provides reward signals for both multimodal understanding and reasoning tasks. Our technical approach comprises two key components: First, we construct a large-scale multimodal preference dataset that covers a wide range of tasks and scenarios, with responses collected from both standard vision-language models (VLMs) and advanced VLM reasoners. Second, we design a reward model architecture based on Qwen2.5-VL-7B-Instruct, integrating a reward head and applying multi-stage fine-tuning using pairwise ranking loss on pairwise preference data. Experimental evaluations show that Skywork-VL Reward achieves state-of-the-art results on multimodal VL-RewardBench and exhibits competitive performance on the text-only RewardBench benchmark. Furthermore, preference data constructed based on our Skywork-VL Reward proves highly effective for training Mixed Preference Optimization (MPO), leading to significant improvements in multimodal reasoning capabilities. Our results underscore Skywork-VL Reward as a significant advancement toward general-purpose, reliable reward models for multimodal alignment. Our model has been publicly released to promote transparency and reproducibility.

Figures (3)

• Figure 1: Distribution of Training Data from Open-Source Sources.
• Figure 2: Evaluating Skywork R1V on Mathematical Problems.
• Figure 3: Evaluating Skywork R1V on Chart Problems.