WorldBench: Disambiguating Physics for Diagnostic Evaluation of World Models

TL;DR

WorldBench introduces a disentangled, video-based benchmark to evaluate physics understanding in world foundation models by separating intuitive physics from physical parameter estimation. The evaluation uses Kubric-generated and real videos, a video completion pipeline, and segmentation-based metrics to quantify both high-level dynamics and low-level parameter accuracy. Empirical results on Cosmos-family WFMs and image-to-video baselines show that outputs are often visually plausible but frequently fail to adhere to underlying physical constants, with high cross-run variance. This framework enables targeted diagnosis of concept-specific failures and supports the development of more physically grounded world-model learning for robust downstream tasks.

Abstract

Recent advances in generative foundational models, often termed "world models," have propelled interest in applying them to critical tasks like robotic planning and autonomous system training. For reliable deployment, these models must exhibit high physical fidelity, accurately simulating real-world dynamics. Existing physics-based video benchmarks, however, suffer from entanglement, where a single test simultaneously evaluates multiple physical laws and concepts, fundamentally limiting their diagnostic capability. We introduce WorldBench, a novel video-based benchmark specifically designed for concept-specific, disentangled evaluation, allowing us to rigorously isolate and assess understanding of a single physical concept or law at a time. To make WorldBench comprehensive, we design benchmarks at two different levels: 1) an evaluation of intuitive physical understanding with concepts such as object permanence or scale/perspective, and 2) an evaluation of low-level physical constants and material properties such as friction coefficients or fluid viscosity. When SOTA video-based world models are evaluated on WorldBench, we find specific patterns of failure in particular physics concepts, with all tested models lacking the physical consistency required to generate reliable real-world interactions. Through its concept-specific evaluation, WorldBench offers a more nuanced and scalable framework for rigorously evaluating the physical reasoning capabilities of video generation and world models, paving the way for more robust and generalizable world-model-driven learning.

Figures (11)

• Figure 1: Overview of our generation and evaluation process. For generation (top), we use Kubric, which uses PyBullet and Blender under the hood. During evaluation (bottom), we first pass the initial frames of the generated video to the world foundation model which completes the video. The completed video is passed to SAM2 along with bounding boxes based on ground truth masks. The segmentations outputted by SAM2 are compared to ground truth segmentations to obtain the final metrics.
• Figure 2: Overview of our physical parameter estimation pipeline. Given an input video, we first use checkerboard detection and SAM2 to extract 3D positions for objects in the video. We then fit curves to these parameters to estimate relevant physical properties such as acceleration or terminal velocity. These are then post-processed, if needed, to calculate the relevant physical parameters.
• Figure 3: Qualitative Examples of the Language-based subset of WorldBench. VLMs are given access to a 9 frame video (same as what is inputted to COSMOS) and ask to answer a True/False or multiple choice question based on the video and future predictions.
• Figure 4: Qualitative examples for the Motion Physics scenario of the intuitive physics subset. For the motion physics example, two objects (a vase and a knot) are thrown at each other, collide, and then fall to the floor.
• Figure 5: Qualitative examples for the Support Relations scenario of the intuitive physics subset A ball is rolled down a ramp towards a solid block near the bottom.