Diffusion-DRF: Differentiable Reward Flow for Video Diffusion Fine-Tuning

TL;DR

Diffusion-DRF tackles the challenge of non-differentiable reward signals in post-training video diffusion by leveraging a frozen Vision-Language Model as a training-free critic and backpropagating its feedback through the diffusion denoising chain to achieve frame- and token-level credit. It introduces a structured VLM feedback pipeline with three facets—Text-Video Alignment (TA), Physical Fidelity (Phy), and Visual-Quality (VQ)—and a differentiable reward flow via a VQA-style loss that updates only the final denoising steps to maintain efficiency. The method eliminates the need to train bespoke reward models or collect large preference datasets while delivering robust improvements in alignment, physics, and perceptual quality across multiple backbones and datasets. Through gradient checkpointing and frame sampling, Diffusion-DRF remains scalable and generalizable to other diffusion-based generation tasks, mitigating reward hacking and model collapse observed with prior approaches.

Abstract

Direct Preference Optimization (DPO) has recently improved Text-to-Video (T2V) generation by enhancing visual fidelity and text alignment. However, current methods rely on non-differentiable preference signals from human annotations or learned reward models. This reliance makes training label-intensive, bias-prone, and easy-to-game, which often triggers reward hacking and unstable training. We propose Diffusion-DRF, a differentiable reward flow for fine-tuning video diffusion models using a frozen, off-the-shelf Vision-Language Model (VLM) as a training-free critic. Diffusion-DRF directly backpropagates VLM feedback through the diffusion denoising chain, converting logit-level responses into token-aware gradients for optimization. We propose an automated, aspect-structured prompting pipeline to obtain reliable multi-dimensional VLM feedback, while gradient checkpointing enables efficient updates through the final denoising steps. Diffusion-DRF improves video quality and semantic alignment while mitigating reward hacking and collapse -- without additional reward models or preference datasets. It is model-agnostic and readily generalizes to other diffusion-based generative tasks.

Figures (10)

• Figure 1: Text-to-video results with Diffusion-DRF. Our method improves both the text-video alignment and physical fidelity of the model, enabling the generation of videos from more challenging prompts..
• Figure 2: Prompting pipeline. Instead of using vague global questions, we propose a prompting pipeline that extracts key points from the prompt (video caption) and formulates questions across three major domains. Each is phrased as a minimal, unambiguous question with a constrained response format, allowing the VLM to answer in binary (Yes/No) with a brief explanation. This targeted questioning reduces ambiguous or uninformative responses and yields per-point alignment signals that can be temporally aggregated into stable supervision for diffusion fine-tuning. We then query the VLM with the same questions on corresponding ground-truth videos to obtain reference answers. Detailed prompt templates and the facets of TA/Phy are provided in the supplementary material.
• Figure 3: Diffusion-DRF Framework. A differentiable video diffusion model fine-tuning paradigm based on a prompting pipeline and a VLM. Diffusion-DRF updates the parameters of the latent model (optionally a DiT DiT) by minimizing the visual question answering loss based on the question and references which include reference answers and reference frames.
• Figure 4: Pair-wise evaluation on VideoGen-Eval. With the same prompt and configuration, we perform pairwise comparisons of generated videos using the VideoAlign scores of text-video alignment (TA), visual quality (VQ), motion quality (MQ) and overall (OA). We compare the Diffusion-DRF with the base model (left) and the Flow-GRPO (right) respectively. A sample is counted as a tie when the absolute difference between the two scores is less than $0.2$.
• Figure 5: Qualitative Comparison. All videos are generated under the same configuration and random seed. The pre-trained model, Flow-GRPO, and the model fine-tuned with VideoAlign all fail to align with the text descriptions. The model trained with PickScore exhibits significant degradation in video quality. Only our model successfully generates videos that accurately satisfy the prompt requirements. In the instance on the left, only our method produces a clear zooming motion. In the instance on the right, only our method correctly generates the grandchild beside the elderly man. Please visit our project page for the full video comparisons between the baselines and our method.