Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving

Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

TL;DR

PHYSICS introduces a challenging university-level physics benchmark with 1,297 expert-annotated problems across six core areas to test deep, multi-step physics reasoning. It pairs a robust automated evaluation pipeline using SymPy and GPT-4o with a broad evaluation of 33 models, revealing substantial gaps between state-of-the-art systems and human experts. The study analyzes prompting strategies, long chain-of-thought reasoning, and retrieval-augmented generation, showing partial gains but no near-domain mastery. The findings highlight critical needs for improved domain knowledge integration and reasoning architectures to advance AI performance on open-ended, expert-level physics problems.

Abstract

We introduce PHYSICS, a comprehensive benchmark for university-level physics problem solving. It contains 1297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. We develop a robust automated evaluation system for precise and reliable validation. Our evaluation of leading foundation models reveals substantial limitations. Even the most advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant challenges in solving high-level scientific problems. Through comprehensive error analysis, exploration of diverse prompting strategies, and Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify key areas for improvement, laying the foundation for future advancements.

PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving

TL;DR

PHYSICS introduces a challenging university-level physics benchmark with 1,297 expert-annotated problems across six core areas to test deep, multi-step physics reasoning. It pairs a robust automated evaluation pipeline using SymPy and GPT-4o with a broad evaluation of 33 models, revealing substantial gaps between state-of-the-art systems and human experts. The study analyzes prompting strategies, long chain-of-thought reasoning, and retrieval-augmented generation, showing partial gains but no near-domain mastery. The findings highlight critical needs for improved domain knowledge integration and reasoning architectures to advance AI performance on open-ended, expert-level physics problems.

Abstract

We introduce PHYSICS, a comprehensive benchmark for university-level physics problem solving. It contains 1297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. We develop a robust automated evaluation system for precise and reliable validation. Our evaluation of leading foundation models reveals substantial limitations. Even the most advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant challenges in solving high-level scientific problems. Through comprehensive error analysis, exploration of diverse prompting strategies, and Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify key areas for improvement, laying the foundation for future advancements.

Paper Structure

This paper contains 40 sections, 81 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Physics Benchmark
  4. Problem Selection
  5. Data Annotation
  6. Data Quality Check
  7. Data Statistics
  8. Experiment
  9. Experimental Setup
  10. Automated Evaluation System
  11. Model Response Processing.
  12. Accuracy Evaluation.
  13. Main Findings
  14. Physics challenges current models.
  15. Proprietary Model Performance Analysis.
  16. ...and 25 more sections

Figures (5)

  • Figure 1: An example of classical mechanics problem in Physics. Physics is a comprehensive benchmark for university-level physics problem solving which contains 1,297 expert-annotated problems.
  • Figure 2: For the overall process, we begin by collecting annotated problems from annotators (§\ref{['Data Annotation']}), followed by validation to create a processed dataset. This dataset is then used to prompt models (§\ref{['Experiment Setup']}). The responses from models undergo regular expression pre-processing and SymPy-based processing before final evaluation using an automated system (§\ref{['Automated Evaluation System']}).
  • Figure 3: Reasoning steps distribution.
  • Figure 4: Comparison between different methods.
  • Figure 5: question image