Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

Zeyu Wang, Xiaogang Li, Peiyao Xiao, Qinhao Kong, Ben Wang, Chengliang Xu, Zichao Chen, Bing Zhao, Hu Wei

Abstract

Breakthroughs in frontier theory often depend on the combination of concrete diagrammatic notations with rigorous logic. While multimodal large language models (MLLMs) show promise in general scientific tasks, current benchmarks often focus on local information extraction rather than the global structural logic inherent in formal scientific notations. In this work, we introduce FeynmanBench, the first benchmark centered on Feynman diagram tasks. It is designed to evaluate AI's capacity for multistep diagrammatic reasoning, which requires satisfying conservation laws and symmetry constraints, identifying graph topology, converting between diagrammatic and algebraic representations, and constructing scattering amplitudes under specific conventions and gauges. To support large-scale and reproducible evaluation, we developed an automated pipeline producing diverse Feynman diagrams along with verifiable topological annotations and amplitude results. Our database spans the electromagnetic, weak, and strong interactions of the Standard Model, encompasses over 100 distinct types and includes more than 2000 tasks. Experiments on state-of-the-art MLLMs reveal systematic failure modes, including unstable enforcement of physical constraints and violations of global topological conditions, highlighting the need for physics-grounded benchmarks for visual reasoning over scientific notation. FeynmanBench provides a logically rigorous test of whether AI can effectively engage in scientific discovery, particularly within theoretical physics.

FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

Abstract

Breakthroughs in frontier theory often depend on the combination of concrete diagrammatic notations with rigorous logic. While multimodal large language models (MLLMs) show promise in general scientific tasks, current benchmarks often focus on local information extraction rather than the global structural logic inherent in formal scientific notations. In this work, we introduce FeynmanBench, the first benchmark centered on Feynman diagram tasks. It is designed to evaluate AI's capacity for multistep diagrammatic reasoning, which requires satisfying conservation laws and symmetry constraints, identifying graph topology, converting between diagrammatic and algebraic representations, and constructing scattering amplitudes under specific conventions and gauges. To support large-scale and reproducible evaluation, we developed an automated pipeline producing diverse Feynman diagrams along with verifiable topological annotations and amplitude results. Our database spans the electromagnetic, weak, and strong interactions of the Standard Model, encompasses over 100 distinct types and includes more than 2000 tasks. Experiments on state-of-the-art MLLMs reveal systematic failure modes, including unstable enforcement of physical constraints and violations of global topological conditions, highlighting the need for physics-grounded benchmarks for visual reasoning over scientific notation. FeynmanBench provides a logically rigorous test of whether AI can effectively engage in scientific discovery, particularly within theoretical physics.

Paper Structure

This paper contains 11 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. FeynmanBench Dataset
  4. Overview
  5. Dataset Generation
  6. Data Classification and Organization
  7. Experiments
  8. Experiment Setup
  9. Results Analysis
  10. Classical Errors
  11. Conclusion and Future Work

Figures (5)

  • Figure 1: The workflow of FeynmanBench. The first four panels handle the generation of Feynman diagrams and associated physical information. The fifth stage applies Feynman rules to derive the corresponding scattering amplitudes. The final stage implements an automated scoring module to evaluate model performance against theoretical ground truth
  • Figure 2: Prompt Engineering and Task Definitions, with Corresponding Solutions and Annotation.
  • Figure 3: Model performance across checkpoints.
  • Figure 4: Heatmaps by model and category for CP1--CP3.
  • Figure 5: illustration of Classical errors