AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization

Abstract

Streaming autoregressive (AR) video generators combined with few-step distillation achieve low-latency, high-quality synthesis, yet remain difficult to align via reinforcement learning from human feedback (RLHF). Existing SDE-based GRPO methods face challenges in this setting: few-step ODEs and consistency model samplers deviate from standard flow-matching ODEs, and their short, low-stochasticity trajectories are highly sensitive to initialization noise, rendering intermediate SDE exploration ineffective. We propose AR-CoPO (AutoRegressive Contrastive Policy Optimization), a framework that adapts the Neighbor GRPO contrastive perspective to streaming AR generation. AR-CoPO introduces chunk-level alignment via a forking mechanism that constructs neighborhood candidates at a randomly selected chunk, assigns sequence-level rewards, and performs localized GRPO updates. We further propose a semi-on-policy training strategy that complements on-policy exploration with exploitation over a replay buffer of reference rollouts, improving generation quality across domains. Experiments on Self-Forcing demonstrate that AR-CoPO improves both out-of-domain generalization and in-domain human preference alignment over the baseline, providing evidence of genuine alignment rather than reward hacking.

Figures (9)

• Figure 1: AR-CoPO is a reinforcement learning for human preference (RLHF) method, aligning few-step autoregressive (AR) video generative models to better sample quality.
• Figure 2: Left: Training curves comparing SDE-based GRPO and AR-CoPO on Self-Forcing. SDE-based GRPO fails to improve the reward, while AR-CoPO consistently achieves higher scores throughout training. Right: Perturbing only the intermediate CM solver noise (Rows 3–5) produces nearly identical outputs, whereas replacing the initial noise (Row 2) causes significant variation, confirming that few-step AR models ( Self-Forcing huang2025selfforcing) are near-deterministic and driven primarily by initial noise.
• Figure 3: The AR-CoPO training pipeline. (1) Rollout: The model autoregressively generates a shared context up to a randomly selected pivot chunk $p$. At chunk $p$, the base initial noise is perturbed into $G$ neighbors; each neighbor is forked into an independent branch and autoregressively completed to produce a full video sequence. (2) Reward: Each completed sequence is decoded and scored by a reward model, yielding a sequence-level reward per branch. (3) Replay & Update: The saved pivot-chunk trajectories are replayed through the current policy; distances between current and old $\hat{x}_0$ predictions induce surrogate policy ratios, which are used in a clipped GRPO update confined to the pivot chunk.
• Figure 4: On-policy semi-on-policy training under AR-CoPO. Left: On-policy training rolls out fresh candidates from the evolving policy $\pi_\theta$ at each iteration, enabling active exploration of new generation modes guided by the reward signal. Right: Semi-on-policy training fixes all rollouts to a reference policy $\pi_{\mathrm{ref}}$; the contrastive objective upweights high-reward candidates and suppresses low-reward ones within a trust region maintained by ratio clipping, enhancing exploitation without sacrificing stability. Each paradigm trains an independent LoRA adapter; merging the two adapters yields the final aligned model that benefits from both exploration and exploitation.
• Figure 5: Qualitative comparison between AR-CoPO (up) and Self-Forcing (down) on diverse text prompts. AR-CoPO produces videos with improved visual fidelity, motion quality, and better adherence to the text prompt.