Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering

TL;DR

This work tackles the challenge of understanding 4D dynamics in video-based reasoning by introducing the DynSuperCLEVR dataset, which emphasizes velocity, acceleration, and collisions, and by proposing NS-4DPhysics, a neural-symbolic model that reconstructs explicit 4D scene representations with physics priors. The approach combines 3D neural mesh-based scene parsing with a probabilistic physical prior and executes reasoning programs to answer diverse questions, including future and counterfactual scenarios. Empirical results show that explicit 4D representations with physics priors substantially outperform previous VideoQA baselines and large multimodal models across factual, predictive, and counterfactual tasks. The work highlights the importance of explicit 4D dynamics for robust physical reasoning and points to future work in extending to real images and more complex objects.

Abstract

For vision-language models (VLMs), understanding the dynamic properties of objects and their interactions in 3D scenes from videos is crucial for effective reasoning about high-level temporal and action semantics. Although humans are adept at understanding these properties by constructing 3D and temporal (4D) representations of the world, current video understanding models struggle to extract these dynamic semantics, arguably because these models use cross-frame reasoning without underlying knowledge of the 3D/4D scenes. In this work, we introduce DynSuperCLEVR, the first video question answering dataset that focuses on language understanding of the dynamic properties of 3D objects. We concentrate on three physical concepts -- velocity, acceleration, and collisions within 4D scenes. We further generate three types of questions, including factual queries, future predictions, and counterfactual reasoning that involve different aspects of reasoning about these 4D dynamic properties. To further demonstrate the importance of explicit scene representations in answering these 4D dynamics questions, we propose NS-4DPhysics, a Neural-Symbolic VideoQA model integrating Physics prior for 4D dynamic properties with explicit scene representation of videos. Instead of answering the questions directly from the video text input, our method first estimates the 4D world states with a 3D generative model powered by physical priors, and then uses neural symbolic reasoning to answer the questions based on the 4D world states. Our evaluation on all three types of questions in DynSuperCLEVR shows that previous video question answering models and large multimodal models struggle with questions about 4D dynamics, while our NS-4DPhysics significantly outperforms previous state-of-the-art models. Our code and data are released in https://xingruiwang.github.io/projects/DynSuperCLEVR/.

Figures (11)

• Figure 1: We propose DynSuperCLEVR to study the 4D dynamics properties of objects and their collisions. We also design three types of questions. Factual questions and counterfactual questions will take the whole 120 frames as input, while the predictive questions will take the first 30 frames.
• Figure 2: An illustration of the construction of DynSuperCLEVR. (a) We use the same 3D object meshes from SuperCLEVR but generate more realistic textures for different colors. (b) The background is created by mapping a real image environment map onto a dome shape. (c) Our video data is fully annotated with 4D dynamic scene structure, containing static and dynamic properties for objects and collision components. (d) For each question, we design new operation programs for 4D dynamic properties, which can be executed on the scene structure to answer the questions.
• Figure 3: Our NS-4DPhysics has three main components. I: A 3D neural mesh scene parser, which combines the rendering likelihood with a physics prior to parse the video into a 4D dynamic scene representation. II: The future states or counterfactual states can be simulated with the reconstruction result by the physics engine. III: A question parser that processes questions into reasoning programs and then executes the program over the predicted scene representation to answer the questions.
• Figure 4: Qualitative examples of factual questions. (a) shows the input video; (b) Our NS-4DPhysics provides a better estimation for the motion with the physical prior. (c) The error in position predicted by the baseline w/o physics in the red box leads to a mistake in the answer.
• Figure 5: Qualitative examples of predictive questions. (a1) The first 30 frames are given to models as input video, and (a2) the following frames are hidden as ground truth future states; (b1) our NS-4DPhysics has a better estimation of the poses of objects, and (b2) provides a plausible simulation by re-simulation. (c1) The red box shows the error in pose estimation of the bus when lacking a physics prior, which (c2) causes the red SUV to collide with the school bus in the future.