Physion: Evaluating Physical Prediction from Vision in Humans and Machines

TL;DR

Physion provides a comprehensive dataset and benchmark to evaluate how well humans and machines predict short-term physical outcomes from realistic, unstructured visual scenes. The study compares diverse model classes—from unsupervised visual dynamics to graph-based, state-aware dynamics—and directly contrasts them with human judgments on an unified Object Contact Prediction task across eight scenarios. Findings show no vision-only model reaches human performance; object-centric representations help but are insufficient, whereas particle- or graph-based models with direct physical state information achieve higher accuracy and human-like patterns. The work emphasizes learning explicit physical representations as the principal bottleneck and provides public data/code to enable reproducible, cross-model transfer benchmarking toward human-like physical understanding.

Abstract

While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.

Figures (10)

• Figure 1: Example frames from the eight Physion scenarios. Red object is agent; yellow is patient.
• Figure 2: Stimulus attributes and task design. (A) Output of TDW for an example frame of a stimulus movie. (B) A schematic of the OCP task: humans and models must predict whether the agent object (red) will contact the patient (yellow), given the initial setup and the motion of the probe (green).
• Figure 3: Human task. (A) Trial structure for the familiarization trials (left) and test trials (right) indicating the Cue, Stimulus, and Inter-trial periods. (B) Example stimuli (rows) including the last frame (not shown during the experiment). Last column indicates the outcome and human accuracy.
• Figure 4: The model benchmarking pipeline including training, architecture, and readout variants.
• Figure 5: Comparisons between humans and models. First row: the all-scenarios trained, observed+simulated-readout task accuracy (A), Pearson correlation between model output and average human response (B), and Cohen's $\kappa$ (C) for each model on each scenario, indicated by its icon. Black icons and the gray zones (2.5th-97.5th percentile) show human performance, mean correlation between split halves of participants, and mean human-human Cohen's $\kappa$, respectively. Second row: accuracy of models across the three readout (D) and training (E) protocols; note that particle-input models have only the observed+simulated readout protocol, as predictions are made based solely on whether two objects came within a threshold distance at the end of the predicted dynamics.