PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation

TL;DR

The paper tackles the problem of physically consistent text-to-video generation, addressing data scarcity and limited physics reasoning in existing methods. It introduces PhyAugPipe to assemble PhyVidGen-135K data using a vision-language model with chain-of-thought reasoning, and PhyGDPO, a physics-aware groupwise direct preference optimization framework built on a Plackett-Luce model with Physics-Guided Rewarding and LoRA-Switch Reference to enable efficient, physics-focused learning. Empirical results show PhyGDPO significantly surpasses state-of-the-art open-source baselines on VideoPhy2 and PhyGenBench, with substantial gains on hard physical actions and higher human preferences in user studies. Collectively, the work advances physically plausible T2V generation without relying on prompt-extending LLMs, enabling more realistic simulations for applications in gaming, robotics, and media production.

Abstract

Recent advances in text-to-video (T2V) generation have achieved good visual quality, yet synthesizing videos that faithfully follow physical laws remains an open challenge. Existing methods mainly based on graphics or prompt extension struggle to generalize beyond simple simulated environments or learn implicit physical reasoning. The scarcity of training data with rich physics interactions and phenomena is also a problem. In this paper, we first introduce a Physics-Augmented video data construction Pipeline, PhyAugPipe, that leverages a vision-language model (VLM) with chain-of-thought reasoning to collect a large-scale training dataset, PhyVidGen-135K. Then we formulate a principled Physics-aware Groupwise Direct Preference Optimization, PhyGDPO, framework that builds upon the groupwise Plackett-Luce probabilistic model to capture holistic preferences beyond pairwise comparisons. In PhyGDPO, we design a Physics-Guided Rewarding (PGR) scheme that embeds VLM-based physics rewards to steer optimization toward physical consistency. We also propose a LoRA-Switch Reference (LoRA-SR) scheme that eliminates memory-heavy reference duplication for efficient training. Experiments show that our method significantly outperforms state-of-the-art open-source methods on PhyGenBench and VideoPhy2. Please check our project page at https://caiyuanhao1998.github.io/project/PhyGDPO for more video results. Our code, models, and data will be released at https://github.com/caiyuanhao1998/Open-PhyGDPO

Figures (7)

• Figure 1: Text-to-video generation on four challenging action categories: (a) gymnastics, (b) soccer, (c) basketball, and (d) glass smashing. When using our post-training method, PhyGDPO, on Wan2.1-T2V-14B wan, the model yields more physically plausible results than OpenAI Sora2 sora and Google Veo3.1 veo3 by generating well-structured human bodies and reasonable physical interactions such as foot kicking the soccer, basketball passing through the hoop, and glass shattering.
• Figure 2: Our physics-augmented video data construction pipeline (PhyAugPipe) first adopts a VLM, Qwen-2.5-72B-Instruct qwen2, following our designed chain-of-thought (CoT) rule in (b) to select text-video data pairs that contain rich physics interactions and phenomena from a large-scale high-quality text-video data pool in (a). Then in (d), we perform action clustering on the filtered data pairs from (c) through the semantics matching via a sentence Transformer sentence_bert. Subsequently, in (e), we adopt a physics-aware VLM, VideoCon-Physics videophy, to evaluate the difficulty of different action categories and then sample the text-video pairs accordingly as the wining cases of our training data for preference optimization.
• Figure 3: Overview of our PhyGDPO framework. The text prompts without physics reasoning extension in (a) are fed into the T2V model. We propose a LoRA-switch reference scheme in (b) to save the GPU memory and increase the training stability. In (c), PhyGDPO is based on the groupwise Plackett-Luce (PL) probabilistic model and adopts a physics-aware VLM, VideoCon-Physics videophy, to reward the DPO training with the real video as the wining sample as it has perfect physics.
• Figure 4: Results of two challenging actions (gymnastics and polo) on VideoPhy2. Our method generates more physically consistent videos, showing coherent, deformation-free gymnastic movements and realistic ball–mallet striking interactions.
• Figure 5: Comparison on two challenging random user-input actions (squash and handspring). Our method generates more physically plausible videos, capturin racket–ball interactions in squash and well-coordinated body motion in handspring.