Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench

TL;DR

VideoScience-Bench addresses the need to evaluate video models not only on perceptual fidelity but also on their ability to reason scientifically about real-world phenomena. The authors introduce VideoScience-Bench (a 160-question, 14-topic benchmark with 103 undergraduate-level concepts) and VideoScience-Judge, a checklist-based, CV-augmented VLM evaluation framework that grounds judgments in observable evidence. Across seven state-of-the-art models, results show strong visual quality and coherence but clear gaps in Phenomenon Congruency and adherence to physical laws, with VideoScience-Judge providing the strongest alignment to human expert judgments. The work delivers a rigorous auto-evaluation paradigm and datasets to spur advances toward video models that faithfully reason about physics and chemistry, beyond surface-level realism.

Abstract

The next frontier for video generation lies in developing models capable of zero-shot reasoning, where understanding real-world scientific laws is crucial for accurate physical outcome modeling under diverse conditions. However, existing video benchmarks are physical commonsense-based, offering limited insight into video models' scientific reasoning capability. We introduce VideoScience-Bench, a benchmark designed to evaluate undergraduate-level scientific understanding in video models. Each prompt encodes a composite scientific scenario that requires understanding and reasoning across multiple scientific concepts to generate the correct phenomenon. The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry. We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions: Prompt Consistency, Phenomenon Congruency, Correct Dynamism, Immutability, and Spatio-Temporal Continuity. Using a VLM-as-a-Judge to assess video generations, we observe strong correlation with human assessments. To the best of our knowledge, VideoScience-Bench is the first benchmark to evaluate video models not only as generators but also as reasoners, requiring their generations to demonstrate scientific understanding consistent with expected physical and chemical phenomena. Our data and evaluation code are available at: \href{https://github.com/hao-ai-lab/VideoScience}{github.com/hao-ai-lab/VideoScience}.

Figures (10)

• Figure 1: VideoScience-Bench Overview.Top: Comparison of physical-commonsense versus scientific-reasoning generations using Sora-2. The first scientific reasoning scenario hinges on recognizing that water’s high specific heat capacity lets it act as a heat sink, thereby preventing immediate balloon rupture. The second example requires reasoning over physical vibration, wave and material properties. Bottom-left: Subcategory frequency of questions From VideoScience-Bench. Note that we are using two-letter code in the pie chart that maps the subcategory provided in the legend. Bottom-right: Expert annotated model performance of seven video models on VideoScience-Bench. The scores serve as the ground truth for our quantitative correlational analysis.
• Figure 2: Overview of the data creation pipeline. Each researcher selects two or more scientific concepts and references relevant educational materials or videos to design a prompt. Prompts undergo peer and model review, followed by model-based quality checking, before being finalized for dataset inclusion.
• Figure 3: Overview of our VLM-as-a-Judge pipeline, which combines checklist-based scoring, key-frame extraction, and evidence aggregation to ensure grounded and traceable video evaluations.
• Figure 4: Comparison of video generation models on VideoScience. Top: Aluminum-iodine reaction testing chemical dynamics. Sora-2 correctly depicts the expected ignition phenomenon, while Hailuo-2.3 fails to generate the reaction. Bottom: Rotating cups with balls testing centrifugal force understanding. Both Sora-2 and Veo-3, the two highest-ranked models in our evaluation, fail to conduct correct experimental procedure and simulate the expected phenomenon accurately. Human Annotation Rating: Prompt Consistency (PCS), Phenomenon Congruency (PCG), Correct Dynamism (CDN), Immutability (IMB), and Spatio-Temporal Coherence (STC).
• Figure 5: Sora-2 generated video of Curved Refraction Gradient, where Sora-2 correctly depicts the setup and the expected phenomenon, despite the visual output being largely static. Human Annotation Rating: Prompt Consistency (PCS), Phenomenon Congruency (PCG), Correct Dynamism (CDN), Immutability (IMB), and Spatio-Temporal Coherence (STC).