Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation

TL;DR

This paper addresses semantic misalignment in GRPO-based post-training for visual generation by modeling reward uncertainty with a semantic prior anchor. It introduces Bayesian Prior-Guided Optimization (BPGO), which combines inter-group Reliability-Adaptive Scaling (RAS) and intra-group Contrastive Reward Transformation (CRT) to reweight and renormalize rewards. The method yields stronger semantic alignment and fidelity across text-to-video, image-to-video, and text-to-image tasks, outperforming GRPO and DanceGRPO with stable convergence. By anchoring optimization to priors and amplifying reliable signals while dampening ambiguous ones, BPGO provides a scalable approach to uncertainty-aware RLHF for high-capacity visual generators.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective and lightweight framework for post-training visual generative models. However, its performance is fundamentally limited by the ambiguity of textual visual correspondence: a single prompt may validly describe diverse visual outputs, and a single image or video may support multiple equally correct interpretations. This many to many relationship leads reward models to generate uncertain and weakly discriminative signals, causing GRPO to underutilize reliable feedback and overfit noisy ones. We introduce Bayesian Prior-Guided Optimization (BPGO), a novel extension of GRPO that explicitly models reward uncertainty through a semantic prior anchor. BPGO adaptively modulates optimization trust at two levels: inter-group Bayesian trust allocation emphasizes updates from groups consistent with the prior while down-weighting ambiguous ones, and intra-group prior-anchored renormalization sharpens sample distinctions by expanding confident deviations and compressing uncertain scores. Across both image and video generation tasks, BPGO delivers consistently stronger semantic alignment, enhanced perceptual fidelity, and faster convergence than standard GRPO and recent variants.

Figures (12)

• Figure 1: Introducing priors helps to address high uncertainty of textual-visual alignment.
• Figure 2: Enhanced GRPO framework incorporating Reliability-Adaptive Scaling (RAS) and Contrastive Reward Transformation (CRT). The architecture features two parallel branches: the RAS branch adaptively reweights samples based on their deviation from prior reward estimates, while the CRT branch generates an auxiliary reward group through reward renormalization to sharpen policy discrimination.
• Figure 3: Visual comparison on text-to-video task. For each videos, the top row is GRPO, and the bottom row is BPGO.
• Figure 4: Visual comparison on image-to-video task. For each videos, the top row is base model, and the bottom row is BPGO.
• Figure 5: Visual comparison on text-to-image task. For each pair, the left is GRPO, and the right is BPGO.