PhyRPR: Training-Free Physics-Constrained Video Generation

TL;DR

Current diffusion-based video models struggle to satisfy explicit physical constraints due to entangled reasoning and rendering. PhyRPR introduces a training-free three-stage pipeline (PhyReason, PhyPlan, PhyRefine) that decouples physical understanding from rendering, using a large multimodal model for physically grounded keyframes, deterministic motion planning for a coarse scaffold, and latent-space fusion to enforce dynamics during diffusion sampling. This approach yields higher physical plausibility and trajectory controllability while maintaining visual fidelity, demonstrated across text-only and image-conditioned tasks with comprehensive baselines and ablative analyses. The proposed framework enables practical physics-constrained video synthesis without expensive training or dedicated physics datasets, broadening feasibility for controllable video generation in real-world scenarios.

Abstract

Recent diffusion-based video generation models can synthesize visually plausible videos, yet they often struggle to satisfy physical constraints. A key reason is that most existing approaches remain single-stage: they entangle high-level physical understanding with low-level visual synthesis, making it hard to generate content that require explicit physical reasoning. To address this limitation, we propose a training-free three-stage pipeline,\textit{PhyRPR}:\textit{Phy\uline{R}eason}--\textit{Phy\uline{P}lan}--\textit{Phy\uline{R}efine}, which decouples physical understanding from visual synthesis. Specifically, \textit{PhyReason} uses a large multimodal model for physical state reasoning and an image generator for keyframe synthesis; \textit{PhyPlan} deterministically synthesizes a controllable coarse motion scaffold; and \textit{PhyRefine} injects this scaffold into diffusion sampling via a latent fusion strategy to refine appearance while preserving the planned dynamics. This staged design enables explicit physical control during generation. Extensive experiments under physics constraints show that our method consistently improves physical plausibility and motion controllability.

Figures (5)

• Figure 1: Samples produced by our method. (a--b) require physical priors, while (c--d) emphasize strict motion constraints. Our three-stage pipeline PhyRPR (PhyReason$\rightarrow$PhyPlan$\rightarrow$PhyRefine) decouples physical reasoning from rendering to better satisfy physical constraints while maintaining high-fidelity video quality.
• Figure 2: Prior video generation models fail to accurately follow the provided physical constraints.
• Figure 3: Overview of our training-free three-stage pipeline PhyRPR: PhyReason, PhyPlan, and PhyRefine. Stage 1 (see \ref{['sec:phyreason']}) outputs physically consistent keyframes and object states. Stage 2 (see \ref{['sec:phyplan']}) uses an LMM to select motion primitives and parameters, and deterministically renders a coarse motion video. Stage 3 (see \ref{['sec:phyrefine']}) applies motion-aware noise-consistent injection(NANC) to enforce the planned kinematics while preserving visual coherence.
• Figure 4: Qualitative comparison with baselines. We use the first frame from PhyReason as the reference for I2V baselines. Our method better captures the physical process, including deformation during inflation and a plausible rebound.
• Figure 5: Qualitative comparison with baselines. We specify physical constraints via text prompts or arrow guidance. Compared to baseline methods, our approach generates videos that more faithfully satisfy the specified physical constraints.