Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Language Models Cannot Reason About Physical Transformation

Dezhi Luo, Yijiang Li, Maijunxian Wang, Tianwei Zhao, Bingyang Wang, Siheng Wang, Pinyuan Feng, Pooyan Rahmanzadehgervi, Ziqiao Ma, Hokin Deng

TL;DR

Findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes, and ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations is introduced.

Abstract

Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with visual content. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.

Vision Language Models Cannot Reason About Physical Transformation

TL;DR

Findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes, and ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations is introduced.

Abstract

Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with visual content. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.
Paper Structure (37 sections, 8 figures, 6 tables)

This paper contains 37 sections, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Evaluating VLMs.
  4. Physical Understanding and Conservation.
  5. Experimental Design
  6. Conservation Tasks
  7. Generalization across Task-irrelevant Features.
  8. Transformation-mandatory vs. -helpful.
  9. Non-conserving Tasks
  10. Adaptation to Multi-frame Input
  11. Temporal Resolution
  12. Sampling Strategy
  13. Experiments
  14. Inference and Evaluation
  15. Inference.
  16. ...and 22 more sections

Figures (8)

  • Figure 1: Illustrative Tasks and Frame Selection Pipeline in Conservation Bench.
  • Figure 2: Overall Performance on ConservationBench. A. Accuracy averaged across conservation tasks and non-conserving control compared to strict pairwise calculation (Top 30 models; full results available in Appendix \ref{['app:complete']}; B. Performance on non-conserving control in relation to conservation tasks.
  • Figure 3: Response patterns under Empty Image and Text-only conditions compared to standard one. We report a distribution change in prediction between Empty Image and standard condition (A) to the left; Text-only control condition and standard condition (B) to the right.
  • Figure 4: Model performance showing main effects by (A) prompt type, (B) number of frames, and (C) frame sampling method. Each panel averages across the other 2 factors from the full factorial design (4 prompts × 5 frame counts × 3 extraction methods).
  • Figure 5: Conservation reasoning does not emerge with model scale. Model performance on (left) conservation tasks shows no relationship with parameter count ($R^2 = 0.019$), while (right) non-conserving task accuracy exhibits only modest scaling effects ($R^2 = 0.239$), both evaluated at 7-frame condition across 112 VLMs.
  • ...and 3 more figures