VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction

TL;DR

VisPhyWorld reframes physical understanding as an executable hypothesis by requiring multimodal models to produce runnable physics-simulation code from visual observations, thereby decoupling appearance from real dynamics and enabling falsifiable, inspectable world models. Through VisPhyBench, a 209-video benchmark spanning 108 physical templates in 2D and 3D, the paper demonstrates that while state-of-the-art MLLMs excel at semantic scene parsing, they often fail to parameterize simple Newtonian dynamics and maintain physically plausible motion when constrained to a fixed physics engine. The approach yields a multi-faceted evaluation that combines reconstruction quality, visual semantics, and motion/physical plausibility, highlighting a dissociation between appearance-based metrics and true physical reasoning. The findings advocate for hybrid representations that ground perception in explicit physical laws to enable transparent, verifiable evaluation of physical understanding in AI systems with potential benefits for robotics and safety-critical deployments.

Abstract

Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% on the benchmark. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics.

Figures (21)

• Figure 1: MLLMs struggle to simulate physical dynamics. Under the same inputs, code generated with rigid-body simulation backends (Three.js/P5.js) produces more physically consistent rollouts, whereas non-physics backends (SVG/Manim) often exhibit implausible motion or contact artifacts such as interpenetration.
• Figure 2: Unlike traditional VQA paradigms, VisPhyWorld accesses physical understanding evaluation by requiring MLLMs to actively reconstruct scenes via executable code, offering superior reasoning explainability compared to traditional paradigms.
• Figure 3: VisPhyWorld Framework.(1) System & Data Construction: We process raw video sequences to extract key frames ($I_{\text{start}}, I_{\text{later}}$) and detection contexts using multimodal agents. (2) Pipeline & Simulation Flow: An LLM-based agent performs motion analysis and generates raw executable code, which is then sanitized and rendered. (3) Evaluation Benchmark: We propose a multi-metric benchmark integrating semantic and physical fidelity to compare generated videos $\hat{X}$ with ground truth $X$. (4) A Detailed Case: A example illustrating how VisPhyWorld translates a collision scene (red ball hits block stack) into executable simulation logic.
• Figure 4: Key metrics on VisPhyBench. We compare code-driven reconstruction (multiple MLLMs) against pixel-space baselines (Veo 3.1 and SVD) under the unified evaluation protocol.
• Figure 5: This case shows that VisPhyWorld exhibits strong physical grounding, correctly simulating the collision dynamics. More examples are in the Appendix.