ComPhy: Compositional Physical Reasoning of Objects and Events from Videos

TL;DR

ComPhy introduces a dataset and benchmark for compositional physical reasoning where intrinsic properties like mass and charge are not visually observable. It shows that standard video reasoning models struggle to infer these hidden properties and predict dynamics from few examples, motivating CPL, an oracle neural-symbolic framework that disentangles perception, property learning, dynamics, and symbolic execution. CPL uses a graph neural network to infer per-object masses $m_i$ and pairwise relative charges $e_{i,j}$, then predicts future dynamics and answers questions through symbolic execution over a parsed program. Results indicate CPL outperforms baselines on factual, predictive, and counterfactual questions, with high per-property inference accuracy ($m_i$ and $e_{i,j}$ near $90\%$), though generalization to more complex scenes remains an open challenge, underscoring the importance of integrating hidden physics with interpretable reasoning for real-world physical understanding.

Abstract

Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not directly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset. For a given set of objects, ComPhy includes few videos of them moving and interacting under different initial conditions. The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions posted on one of the videos. Evaluation results of several state-of-the-art video reasoning models on ComPhy show unsatisfactory performance as they fail to capture these hidden properties. We further propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual perception, physical property learning, dynamic prediction, and symbolic execution into a unified framework. CPL can effectively identify objects' physical properties from their interactions and predict their dynamics to answer questions.

Figures (9)

• Figure 1: Non-visual properties like mass and charge govern the interaction between objects and lead to different motion trajectories. a) Objects attract and repel each other according to the (sign of) charge they carry. b) Mass determines how much an object's trajectory is perturbed during an interaction. Heavier objects have more stable motion.
• Figure 2: Sample target video, reference videos and question-answer pairs from ComPhy.
• Figure 3: The perception module detects objects' location and visual appearance attributes. The physical property learner learns objects' properties based on detected object trajectories. The dynamic predictor predicts objects' dynamics in the counterfactual scene based on objects' properties and locations. Finally, an execution engine runs the program parsed by the language parser on the predicted dynamic scene to answer the question.
• Figure 3: Evaluation of CPL on ComPhy.
• Figure 4: Generalization of physical reasoning.