Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Opinion: Learning Intuitive Physics May Require More than Visual Data

Ellen Su, Solim Legris, Todd M. Gureckis, Mengye Ren

TL;DR

The paper tests whether data distribution, not just volume, enables learning intuitive physics in video models by pretraining V-JEPA on developmentally realistic Egocentric SAYCam data. Across IntPhys2, models trained on SAYCam or large internet datasets perform near chance, suggesting data volume and distribution alone are insufficient for current architectures to acquire intuitive physics. The authors analyze model 'surprise' signals and find no clear advantage from developmentally realistic data, underscoring the need for embodied, multimodal signals and architectural innovations to realize artificial intuitive physics. They propose leveraging action, motion, and accelerometer data (embodiment) in future datasets and methods to better capture physical principles.

Abstract

Humans expertly navigate the world by building rich internal models founded on an intuitive understanding of physics. Meanwhile, despite training on vast quantities of internet video data, state-of-the-art deep learning models still fall short of human-level performance on intuitive physics benchmarks. This work investigates whether data distribution, rather than volume, is the key to learning these principles. We pretrain a Video Joint Embedding Predictive Architecture (V-JEPA) model on SAYCam, a developmentally realistic, egocentric video dataset partially capturing three children's everyday visual experiences. We find that training on this dataset, which represents 0.01% of the data volume used to train SOTA models, does not lead to significant performance improvements on the IntPhys2 benchmark. Our results suggest that merely training on a developmentally realistic dataset is insufficient for current architectures to learn representations that support intuitive physics. We conclude that varying visual data volume and distribution alone may not be sufficient for building systems with artificial intuitive physics.

Opinion: Learning Intuitive Physics May Require More than Visual Data

TL;DR

The paper tests whether data distribution, not just volume, enables learning intuitive physics in video models by pretraining V-JEPA on developmentally realistic Egocentric SAYCam data. Across IntPhys2, models trained on SAYCam or large internet datasets perform near chance, suggesting data volume and distribution alone are insufficient for current architectures to acquire intuitive physics. The authors analyze model 'surprise' signals and find no clear advantage from developmentally realistic data, underscoring the need for embodied, multimodal signals and architectural innovations to realize artificial intuitive physics. They propose leveraging action, motion, and accelerometer data (embodiment) in future datasets and methods to better capture physical principles.

Abstract

Humans expertly navigate the world by building rich internal models founded on an intuitive understanding of physics. Meanwhile, despite training on vast quantities of internet video data, state-of-the-art deep learning models still fall short of human-level performance on intuitive physics benchmarks. This work investigates whether data distribution, rather than volume, is the key to learning these principles. We pretrain a Video Joint Embedding Predictive Architecture (V-JEPA) model on SAYCam, a developmentally realistic, egocentric video dataset partially capturing three children's everyday visual experiences. We find that training on this dataset, which represents 0.01% of the data volume used to train SOTA models, does not lead to significant performance improvements on the IntPhys2 benchmark. Our results suggest that merely training on a developmentally realistic dataset is insufficient for current architectures to learn representations that support intuitive physics. We conclude that varying visual data volume and distribution alone may not be sufficient for building systems with artificial intuitive physics.

Paper Structure

This paper contains 12 sections, 2 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Pretraining V-JEPA on a developmentally realistic dataset
  3. SAYCam dataset
  4. Evaluating intuitive physics
  5. Implementation details
  6. V-JEPA pretrained on SAYCam yields similar performance to SOTA models
  7. Surprise analysis
  8. Discussion
  9. SAYCam dataset
  10. Violation of Expectation Metric
  11. Benchmark details
  12. Implementation Details

Figures (2)

  • Figure 1: Breakdown of model accuracies on the IntPhys2 benchmark by physical condition, difficulty, and camera set up. All models achieve around chance performance at classifying between possible and impossible physical events with small variations in rank. Error bars mark standard error of the mean accuracy.
  • Figure 2: Model surprises for all V-JEPA models. A) Top: a subsample of video frames for an impossible video (violates object permanence). Bottom: Frame-by-frame surprise predictions for the corresponding possible/impossible video pair which all models classified correctly (average surprise for possible < average surprise for impossible). B) Top: a subsample of video frames for an impossible video (violates continuity). Bottom: Frame-by-frame surprise predictions for the corresponding possible/impossible video pair which all models classified incorrectly (average surprise for possible > average surprise for impossible). The surprise values were taken from the context which reported the best overall accuracy score. Dotted lines represent average surprise scores.