Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RealDPO: Real or Not Real, that is the Preference

Guo Cheng, Danni Yang, Ziqi Huang, Jianlou Si, Chenyang Si, Ziwei Liu

TL;DR

This work tackles the difficulty of producing natural, contextually coherent human motion in video generation by proposing RealDPO, a real-data–driven Direct Preference Optimization workflow that avoids reward-model pitfalls. By treating diffusion-based video generation as a diffusion MDP and optimizing with win-lose preferences where real videos serve as wins, RealDPO achieves stronger visual-text alignment and motion realism with a data-efficient approach. It introduces RealAction-5K, a compact, high-quality dataset of daily actions to guide preference learning, and employs a tailored loss with an EMA-updated reference model to iteratively refine outputs. Empirical results across human studies and automated metrics show RealDPO outperforming SFT, LiFT, VideoAlign, and reward-model–based methods, underscoring its practical impact for action-centric video synthesis and its potential generalization to broader domains.

Abstract

Video generative models have recently achieved notable advancements in synthesis quality. However, generating complex motions remains a critical challenge, as existing models often struggle to produce natural, smooth, and contextually consistent movements. This gap between generated and real-world motions limits their practical applicability. To address this issue, we introduce RealDPO, a novel alignment paradigm that leverages real-world data as positive samples for preference learning, enabling more accurate motion synthesis. Unlike traditional supervised fine-tuning (SFT), which offers limited corrective feedback, RealDPO employs Direct Preference Optimization (DPO) with a tailored loss function to enhance motion realism. By contrasting real-world videos with erroneous model outputs, RealDPO enables iterative self-correction, progressively refining motion quality. To support post-training in complex motion synthesis, we propose RealAction-5K, a curated dataset of high-quality videos capturing human daily activities with rich and precise motion details. Extensive experiments demonstrate that RealDPO significantly improves video quality, text alignment, and motion realism compared to state-of-the-art models and existing preference optimization techniques.

RealDPO: Real or Not Real, that is the Preference

TL;DR

This work tackles the difficulty of producing natural, contextually coherent human motion in video generation by proposing RealDPO, a real-data–driven Direct Preference Optimization workflow that avoids reward-model pitfalls. By treating diffusion-based video generation as a diffusion MDP and optimizing with win-lose preferences where real videos serve as wins, RealDPO achieves stronger visual-text alignment and motion realism with a data-efficient approach. It introduces RealAction-5K, a compact, high-quality dataset of daily actions to guide preference learning, and employs a tailored loss with an EMA-updated reference model to iteratively refine outputs. Empirical results across human studies and automated metrics show RealDPO outperforming SFT, LiFT, VideoAlign, and reward-model–based methods, underscoring its practical impact for action-centric video synthesis and its potential generalization to broader domains.

Abstract

Video generative models have recently achieved notable advancements in synthesis quality. However, generating complex motions remains a critical challenge, as existing models often struggle to produce natural, smooth, and contextually consistent movements. This gap between generated and real-world motions limits their practical applicability. To address this issue, we introduce RealDPO, a novel alignment paradigm that leverages real-world data as positive samples for preference learning, enabling more accurate motion synthesis. Unlike traditional supervised fine-tuning (SFT), which offers limited corrective feedback, RealDPO employs Direct Preference Optimization (DPO) with a tailored loss function to enhance motion realism. By contrasting real-world videos with erroneous model outputs, RealDPO enables iterative self-correction, progressively refining motion quality. To support post-training in complex motion synthesis, we propose RealAction-5K, a curated dataset of high-quality videos capturing human daily activities with rich and precise motion details. Extensive experiments demonstrate that RealDPO significantly improves video quality, text alignment, and motion realism compared to state-of-the-art models and existing preference optimization techniques.
Paper Structure (25 sections, 7 equations, 10 figures, 3 tables)

This paper contains 25 sections, 7 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Denoising Process as Multi-Step MDP
  5. DPO for Diffusion MDP
  6. The RealDPO Paradigm
  7. RealAction: Preference Data Collection
  8. Win-Lose Sampling for DPO Training
  9. Preference Learning for Video Generation
  10. Experiments
  11. Quantitative Comparisons
  12. Qualitative Comparisons
  13. Conclusion
  14. Acknowledgements
  15. More Qualitative Results
  16. ...and 10 more sections

Figures (10)

  • Figure 1: Can we align video generative models using real data as preference data without a reward model? (a) Comparison between using the reward model to score synthetic data for preference learning and our RealDPO method, which uses high-quality real data as win samples. Our method avoids the limitations of the reward model and the associated hacking issues. (b) Comparison between the video generated by the pretrained model and the real video for the same scene. The three scores on the right represent the scores given by the reward model from VisionReward xu2024visionreward, the human action metric from VBench huang2024vbenchhuang2024vbench++, and human preference, respectively. It can be observed that while the existing reward model and VBench can evaluate semantic correctness, they are limited in assessing human motion quality. (c) Three model-generated examples from the same prompt, each with different initial noise, exhibit poor limb interaction, making it challenging for human annotators to identify which sample should be chosen as the win sample for reward model training.
  • Figure 2: RealDPO vs SFT. A qualitative comparison between RealDPO and supervised fine-tuning (SFT). RealDPO demonstrates more natural motion generation. For more details regarding the comparison, please refer to the supplementary material.
  • Figure 3: Overview of the RealAction-5K Dataset. (a) Samples of RealAction-5K Dataset (b) Data Collection and Processing Pipeline (c) Video Caption Word Distribution (d) Action Content Distribution (e) Prompt Length Distribution
  • Figure 4: The RealDPO Framework. We use real data as the win samples in DPO, and illustrate the data pipeline on the left hand side. We present the RealDPO loss, and reference model update strategy on the right hand side.
  • Figure 5: Qualitative Results. We visualize the effect of before and after applying RealDPO. See the supplementary for videos.
  • ...and 5 more figures