Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhyEduVideo: A Benchmark for Evaluating Text-to-Video Models for Physics Education

Megha Mariam K. M, Aditya Arun, Zakaria Laskar, C. V. Jawahar

TL;DR

PhyEduVideo introduces a pedagogy-centered benchmark to evaluate text-to-video models for physics education, grounding assessment in teaching-point prompts that map to core concepts. The framework measures Semantic Alignment, Physics Commonsense, Motion Smoothness, and Temporal Flickering, validated by a human study showing stronger alignment with human judgments than prior benchmarks. Empirical results reveal a persistent gap between visual realism and conceptual accuracy, with Wan2.1 and PhyT2V providing the best educational performance, especially in mechanics, fluids, and optics, while electromagnetism and thermodynamics remain challenging. The work offers a valuable resource and baseline for developing physics-aware T2V systems capable of generating curriculum-aligned educational content at scale, and the accompanying codebase is publicly available.

Abstract

Generative AI models, particularly Text-to-Video (T2V) systems, offer a promising avenue for transforming science education by automating the creation of engaging and intuitive visual explanations. In this work, we take a first step toward evaluating their potential in physics education by introducing a dedicated benchmark for explanatory video generation. The benchmark is designed to assess how well T2V models can convey core physics concepts through visual illustrations. Each physics concept in our benchmark is decomposed into granular teaching points, with each point accompanied by a carefully crafted prompt intended for visual explanation of the teaching point. T2V models are evaluated on their ability to generate accurate videos in response to these prompts. Our aim is to systematically explore the feasibility of using T2V models to generate high-quality, curriculum-aligned educational content-paving the way toward scalable, accessible, and personalized learning experiences powered by AI. Our evaluation reveals that current models produce visually coherent videos with smooth motion and minimal flickering, yet their conceptual accuracy is less reliable. Performance in areas such as mechanics, fluids, and optics is encouraging, but models struggle with electromagnetism and thermodynamics, where abstract interactions are harder to depict. These findings underscore the gap between visual quality and conceptual correctness in educational video generation. We hope this benchmark helps the community close that gap and move toward T2V systems that can deliver accurate, curriculum-aligned physics content at scale. The benchmark and accompanying codebase are publicly available at https://github.com/meghamariamkm/PhyEduVideo.

PhyEduVideo: A Benchmark for Evaluating Text-to-Video Models for Physics Education

TL;DR

PhyEduVideo introduces a pedagogy-centered benchmark to evaluate text-to-video models for physics education, grounding assessment in teaching-point prompts that map to core concepts. The framework measures Semantic Alignment, Physics Commonsense, Motion Smoothness, and Temporal Flickering, validated by a human study showing stronger alignment with human judgments than prior benchmarks. Empirical results reveal a persistent gap between visual realism and conceptual accuracy, with Wan2.1 and PhyT2V providing the best educational performance, especially in mechanics, fluids, and optics, while electromagnetism and thermodynamics remain challenging. The work offers a valuable resource and baseline for developing physics-aware T2V systems capable of generating curriculum-aligned educational content at scale, and the accompanying codebase is publicly available.

Abstract

Generative AI models, particularly Text-to-Video (T2V) systems, offer a promising avenue for transforming science education by automating the creation of engaging and intuitive visual explanations. In this work, we take a first step toward evaluating their potential in physics education by introducing a dedicated benchmark for explanatory video generation. The benchmark is designed to assess how well T2V models can convey core physics concepts through visual illustrations. Each physics concept in our benchmark is decomposed into granular teaching points, with each point accompanied by a carefully crafted prompt intended for visual explanation of the teaching point. T2V models are evaluated on their ability to generate accurate videos in response to these prompts. Our aim is to systematically explore the feasibility of using T2V models to generate high-quality, curriculum-aligned educational content-paving the way toward scalable, accessible, and personalized learning experiences powered by AI. Our evaluation reveals that current models produce visually coherent videos with smooth motion and minimal flickering, yet their conceptual accuracy is less reliable. Performance in areas such as mechanics, fluids, and optics is encouraging, but models struggle with electromagnetism and thermodynamics, where abstract interactions are harder to depict. These findings underscore the gap between visual quality and conceptual correctness in educational video generation. We hope this benchmark helps the community close that gap and move toward T2V systems that can deliver accurate, curriculum-aligned physics content at scale. The benchmark and accompanying codebase are publicly available at https://github.com/meghamariamkm/PhyEduVideo.
Paper Structure (15 sections, 13 figures, 3 tables)

This paper contains 15 sections, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Video Models
  4. Evaluation Benchmarks for Text-to-Video Models
  5. PhyEduVideo
  6. Metrics
  7. Human Evaluation
  8. Experiments
  9. Evaluated Models
  10. Quantitative Evaluation
  11. Qualitative Evaluation
  12. Conclusion
  13. Model Details
  14. Human Evaluation
  15. Analysis of Score Mismatches Between PhyEduVideo and Human Evaluators

Figures (13)

  • Figure 1: Overview of the PhyEduVideo Benchmark.The construction pipeline, from concept extraction to prompt generation. Concept distribution across five major physics domains: Mechanics, Electromagnetism (EM), Optics, Thermodynamics (Thermo), Fluids, and Waves & Oscillations ( W&O). Standardized representation of each concept, detailing key teaching points and corresponding video prompts. Example concepts with teaching points, visual prompts, and representative generated video frames. As shown, current T2V models often fail to produce videos that are both semantically aligned and physically plausible—for example, in T04 (Relative Motion), the two toy cars were intended to move side by side at the same speed, but the generated video deviates from this.
  • Figure 2: Overview of benchmark statistics for the PhyEduVideo dataset. (a) Distribution of teaching points across physics concepts, (b) Word cloud of frequent prompt terms, (c) Distribution of prompt lengths.
  • Figure 3: Comparison of SA (Semantic Adherence) and PC (Physics Commonsense) scores assigned by the VideoPhy, Automatic Evaluator (PhyEduVideo) and humans. Detailed videos are available on the GitHub page.
  • Figure 4: Qualitative comparisons of generated videos across six classical physics categories—Mechanics, Waves & Oscillations, Fluids, Thermodynamics, Electromagnetism, and Optics—for five T2V models: VideoCrafter2, CogVideoX, Wan2.1, Video-MSG, and PhyT2V. Detailed videos are available on the GitHub page.
  • Figure 5: Guidelines and rules given to human annotators to ensure consistent and reliable evaluation.
  • ...and 8 more figures