Growing with the Generator: Self-paced GRPO for Video Generation

TL;DR

This work tackles the problem of static reward models in reinforcement learning for video generation by introducing Self-Paced GRPO, a competence-aware framework where the reward signal co-evolves with the generator. The method defines a latent, multi-term reward with adaptive weights $w_j$ governed by a continuous curriculum through a three-stage progression: visual fidelity, temporal coherence, and text–video alignment, driven by a competence function $c$ and a transition function $g_j$. Empirically, Self-Paced GRPO yields consistent improvements across backbones (Wan2.1-T2V and HunyuanVideo) and reward models (including large VLMs like $\text{Qwen}_{2.5\text{VL}}-72\text{B}$) on VBench, while reducing reward bias and instability relative to fixed-reward baselines. The results demonstrate that adaptive reward learning can provide stable, scalable reinforcement-based alignment for high-dimensional generative video tasks, with potential for broader curriculum-based reward design in multimodal generation. Overall, the paper offers a practical, generalizable framework for evolving supervisory signals in post-training video generation and suggests promising directions for future research in curriculum learning and multi-objective RL in vision–language contexts.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as a powerful reinforcement learning paradigm for post-training video generation models. However, existing GRPO pipelines rely on static, fixed-capacity reward models whose evaluation behavior is frozen during training. Such rigid rewards introduce distributional bias, saturate quickly as the generator improves, and ultimately limit the stability and effectiveness of reinforcement-based alignment. We propose Self-Paced GRPO, a competence-aware GRPO framework in which reward feedback co-evolves with the generator. Our method introduces a progressive reward mechanism that automatically shifts its emphasis from coarse visual fidelity to temporal coherence and fine-grained text-video semantic alignment as generation quality increases. This self-paced curriculum alleviates reward-policy mismatch, mitigates reward exploitation, and yields more stable optimization. Experiments on VBench across multiple video generation backbones demonstrate consistent improvements in both visual quality and semantic alignment over GRPO baselines with static rewards, validating the effectiveness and generality of Self-Paced GRPO.

Figures (14)

• Figure 1: As video quality improves, scores of static reward models may flatten and fail to capture group-level discrimination. In contrast, evolving rewards adapt to the generation competence and remain sensitive to quality gains throughout training.
• Figure 2: Illustration of reward model bias. VideoAlign shows content-specific preference, often assigning higher scores to certain video categories.
• Figure 3: Pipeline of the proposed Self-Paced GRPO. The generator produces candidate outputs, which are evaluated by a progressive co-evolving reward mechanism. Rewards are aggregated and gated through a competence-aware adaptation strategy, enabling adaptive stage-wise aggregation. The normalized reward feedbacks are then used to compute advantages that guide gradient-based fine-tuning of the generator.
• Figure 4: Reward preference analysis. We partition 2,000 real videos into 20 semantic categories, evaluate them using different reward models, and report the coefficient of variation of the mean scores across categories.
• Figure 5: Ablation study: we further grouped 2,000 iStock istock2025video videos into 10--20 semantic clusters, using VideoAlignliu2025improving and Qwen2.5VL-72B to evaluate them, and compute the coefficient of variation of the mean reward scores within each cluster.