TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment

TL;DR

TAGRPO addresses the limited success of applying visual GRPO to image-to-video diffusion by introducing a trajectory-alignment objective and a memory bank to exploit inter-sample relations among rollout videos. The method defines a trajectory alignment loss that encourages all samples in a group to follow the high-reward trajectory and diverge from the low-reward one, combined with standard GRPO into J_TAGRPO = J_GRPO + gamma J_align; a memory bank stores past latents and rewards to improve efficiency. Experiments on Wan 2.2 and HunyuanVideo-1.5 show TAGRPO achieving faster convergence and higher rewards than DanceGRPO across 320p and 720p and with different reward models, demonstrating strong generalization. The contributions include a novel trajectory-alignment framework, a memory-bank mechanism to reduce rollout costs, and demonstrated state-of-the-art GRPO-based post-training for I2V, with plans to open-source code and models.

Abstract

Recent studies have demonstrated the efficacy of integrating Group Relative Policy Optimization (GRPO) into flow matching models, particularly for text-to-image and text-to-video generation. However, we find that directly applying these techniques to image-to-video (I2V) models often fails to yield consistent reward improvements. To address this limitation, we present TAGRPO, a robust post-training framework for I2V models inspired by contrastive learning. Our approach is grounded in the observation that rollout videos generated from identical initial noise provide superior guidance for optimization. Leveraging this insight, we propose a novel GRPO loss applied to intermediate latents, encouraging direct alignment with high-reward trajectories while maximizing distance from low-reward counterparts. Furthermore, we introduce a memory bank for rollout videos to enhance diversity and reduce computational overhead. Despite its simplicity, TAGRPO achieves significant improvements over DanceGRPO in I2V generation.

Figures (6)

• Figure 1: Performance of the proposed TAGRPO. We mainly compared our method with DanceGRPO xue2025dancegrpo, as existing open‑sourced implementations of visual GRPO methods liu2025flowhe2025tempflowzheng2025diffusionnft typically support text‑conditioned tasks, with DanceGRPO being the only exception. The results demonstrate that TAGRPO achieved faster convergence and consistently higher reward gains on both Wan 2.2 wan2025wan and HunyuanVideo-1.5 wu2025hunyuanvideo. We used Q‑Save wu2025q and HPSv3 ma2025hpsv3 as reward models, and all reported reward values were averaged over the evaluation set.
• Figure 2: Overview of our proposed TAGRPO. Given a training sample, we generate multiple video samples and evaluate them using a reward model. For each group of samples generated from the same initial noise, we apply both the standard GRPO loss and our trajectory-wise loss $\mathcal{J}_{\text{align}}$ on intermediate latents. $\mathcal{J}_{\text{align}}$ implicitly encourages alignment with high-reward trajectories while maintaining distance from low-reward ones. A memory bank stores historical samples and their rewards, enabling efficient exploitation of diverse past generations without requiring large per-step rollouts. For simplicity, we omit the reference model for computing KL divergence.
• Figure 3: Qualitative comparison among TAGRPO, DanceGRPO and the base model Wan 2.2. Models trained with TAGRPO demonstrate superior visual quality with improved aesthetics, reduced distortion artifacts and better motion realism in animation scenes.
• Figure 4: Qualitative comparison among TAGRPO, DanceGRPO and the base model HunyuanVideo 1.5 (HY-1.5). Models trained with TAGRPO exhibit superior generation fidelity, characterized by sharper structural details and significantly fewer temporal artifacts.
• Figure 5: Ablation study on the contributions of $\mathcal{J}_{\text{align}}$ loss and memory bank mechanism. TAGRPO achieves the highest reward improvement, while removing either component results in slower convergence and lower final performance.