Video Generation Models Are Good Latent Reward Models

TL;DR

<3-5 sentence high-level summary>PRFL addresses the challenge of aligning video generation with human preferences by shifting reward modeling from pixel-space to latent space, leveraging pre-trained video generation models to extract motion cues at arbitrary diffusion timesteps. It introduces a two-stage framework: a Process-Aware Video Reward Model (PAVRM) for latent, timestep-aware quality prediction and Process Reward Feedback Learning (PRFL) to fine-tune generators through latent-space rewards with random timestep sampling, avoiding costly VAE decoding. The method achieves substantial improvements in motion-related metrics and human judgments while reducing memory usage and training time compared to RGB ReFL, and it generalizes across Text-to-Video and Image-to-Video tasks. These results demonstrate the practicality of latent, process-level supervision for video generation and open paths for more efficient preference alignment in video synthesis.</$>`

Abstract

Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.

Figures (7)

• Figure 1: PRFL generates high-quality videos with enhanced motion quality. We show results on image-to-video (frames of different sizes) and text-to-video (frames of same size) generation tasks, showing the representative frames from each video.
• Figure 2: Comparison between RGB ReFL and our PRFL. RGB reward models require near-complete denoising and VAE decoding to RGB space, introducing evaluation delay, GPU memory bottleneck, and insufficient supervision of early denoising stages where structure and motion are formed. PRFL eliminates these limitations by performing reward modeling directly in latent space with timestep-aware training.
• Figure 3: Analysis on VGM Features. (a) VLM-based reward model (VideoAlign-MQ) exhibits poor timestep generalization with fluctuating scores. (b) VGM features from any DiT layer uniformly achieve 78.8% accuracy, matching VLM baseline. (c) Timestep-aware fine-tuning unlocks VGM's full potential, achieving 85.46% accuracy with peak performance at early timesteps (t=0.8).
• Figure 4: Overview of our process-aware video generation alignment framework. Left: Architecture of the Video Generation Model (VGM) and Process-Aware Video Reward Model (PAVRM). Right: Two-stage training pipeline. PAVRM training with reward prediction from noisy latents. Process Reward Feedback Learning (PRFL) optimizing VGM through latent-space reinforcement learning at randomly sampled timesteps.
• Figure 5: Human evaluation of PRFL model vs. other post-training methods.