Unified Personalized Reward Model for Vision Generation
Yibin Wang, Yuhang Zang, Feng Han, Jiazi Bu, Yujie Zhou, Cheng Jin, Jiaqi Wang
TL;DR
UnifiedReward-Flex tackles the misalignment of multimodal reward models by introducing context-adaptive, hierarchical evaluation that instantiates prompt-specific criteria. It couples a two-stage training pipeline—reasoning distillation for supervised fine-tuning and reasoning-aware direct preference optimization—with integration into Group Relative Policy Optimization for image and video synthesis. Empirical results show consistent improvements in both semantic alignment and visual quality across diverse in-domain and out-of-domain benchmarks, outperforming fixed-scorer, Bradley-Terry, and static VLM-based baselines. The approach offers practical benefits for reward-driven vision generation, enabling more robust, context-aware optimization in complex prompts and dynamic content domains.
Abstract
Recent advancements in multimodal reward models (RMs) have significantly propelled the development of visual generation. Existing frameworks typically adopt Bradley-Terry-style preference modeling or leverage generative VLMs as judges, and subsequently optimize visual generation models via reinforcement learning. However, current RMs suffer from inherent limitations: they often follow a one-size-fits-all paradigm that assumes a monolithic preference distribution or relies on fixed evaluation rubrics. As a result, they are insensitive to content-specific visual cues, leading to systematic misalignment with subjective and context-dependent human preferences. To this end, inspired by human assessment, we propose UnifiedReward-Flex, a unified personalized reward model for vision generation that couples reward modeling with flexible and context-adaptive reasoning. Specifically, given a prompt and the generated visual content, it first interprets the semantic intent and grounds on visual evidence, then dynamically constructs a hierarchical assessment by instantiating fine-grained criteria under both predefined and self-generated high-level dimensions. Our training pipeline follows a two-stage process: (1) we first distill structured, high-quality reasoning traces from advanced closed-source VLMs to bootstrap SFT, equipping the model with flexible and context-adaptive reasoning behaviors; (2) we then perform direct preference optimization (DPO) on carefully curated preference pairs to further strengthen reasoning fidelity and discriminative alignment. To validate the effectiveness, we integrate UnifiedReward-Flex into the GRPO framework for image and video synthesis, and extensive results demonstrate its superiority.