Morpheus: Benchmarking Physical Reasoning of Video Generative Models with Real Physical Experiments

TL;DR

Morpheus tackles the gap between perceptual realism and physical reasoning in video generative models by grounding evaluation in real-world physics experiments and physics-informed metrics. It maps videos to standardized physical representations and uses PINNs to assess adherence to governing equations and conserved quantities, yielding two main scores: Dynamical and Physical Invariance. The study shows that even state-of-the-art VGMs achieve high visual fidelity but struggle to respect physical laws, with real videos setting the upper bound. By releasing datasets, metrics, and a public leaderboard, Morpheus enables reproducible benchmarking and progress toward genuine world-model capabilities in video generation.

Abstract

Recent advances in image and video generation raise hopes that these models possess world modeling capabilities, the ability to generate realistic, physically plausible videos. This could revolutionize applications in robotics, autonomous driving, and scientific simulation. However, before treating these models as world models, we must ask: Do they adhere to physical conservation laws? To answer this, we introduce Morpheus, a benchmark for evaluating video generation models on physical reasoning. It features 80 real-world videos capturing physical phenomena, guided by conservation laws. Since artificial generations lack ground truth, we assess physical plausibility using physics-informed metrics evaluated with respect to infallible conservation laws known per physical setting, leveraging advances in physics-informed neural networks and vision-language foundation models. Our findings reveal that even with advanced prompting and video conditioning, current models struggle to encode physical principles despite generating aesthetically pleasing videos. All data, leaderboard, and code are open-sourced at our project page.

Figures (27)

• Figure 1: Comparison of evaluation methods for video generative models. a) Human or VLM-based judgments provide only qualitative and subjective assessments of physical plausibility. b) Trajectory matching compares generated and ground-truth paths but may misclassify physically valid trajectories. For example, for projectile motion, many parabolic trajectories are physically plausible when VGMs are conditioned only on an image, as object velocity cannot be estimated from it. c) Our proposed framework, Morpheus, evaluates generated videos via physics-informed scores, testing both conservation of physical invariants and consistency with governing equations of motion.
• Figure 2: The overview of Morpheus benchmark. Video augmentation and generation (upper) and the trajectory extraction pipelines (lower). We start with augmenting recorded videos with realistic style transfer, based on object masks. Next, we use the first frame (or multiple frames in case of video conditioning) of the obtained videos, as well as the textual description, as a prompt for a VGM. After this, we extract object trajectories for both real-world and generated videos using the trajectory extraction pipeline, including trajectory tracking and discarding unreliable trajectories. Finally, we evaluate Morpheus scores for all videos with valid trajectories.
• Figure 3: Examples of physical experiments included in the Morpheus benchmark, illustrating both different dynamics and variations in object types. Top row (left to right): falling ball, projectile motion, holonomic pendulum, sliding, falling apple, and double pendulum. Bottom row (left to right): collision, falling tape, rolling can, spring, rolling orange, and bouncing.
• Figure 4: Evaluation of trajectories extracted from real and VGMs videos using our Dynamical (upper) and Physical Invariance (lower) scores. For the Dynamical score, trajectories from real-world or generated videos are fitted to a PINN with the corresponding equation of motion for the particular physical law as an extra loss term. For the Physical Invariance score, using the same trajectories, we estimate physical quantities that should be invariant in the systems, such as total energy and oscillation period, and use their variance as a measure of physical plausibility.
• Figure 5: Different types of discarded generated videos: (left) A video showing the disappearance of the orange ball during fall; (middle) A video illustrating generation of multiple objects in pendulum experiment; (right) A video in which the double pendulum does not move.