Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, Indranil Das, Junyi Cao, Yufeng Du, Jiabin Yu, Peixue Wu, Jinchen He, Yifan Su, Yikun Jiang, Yujie Zhang, Chang Liu, Ze-Min Huang, Weizhen Jia, Yunkai Wang, Farshid Jafarpour, Yong Zhao, Xinan Chen, Jessie Shelton, Aaron W. Young, John Bartolotta, Wenchao Xu, Yue Sun, Anjun Chu, Victor Colussi, Chris Akers, Nathan Brooks, Wenbo Fu, Jinchao Zhao, Marvin Qi, Anqi Mu, Yubo Yang, Allen Zang, Yang Lyu, Peizhi Mai, Christopher Wilson, Xuefei Guo, Juntai Zhou, Daniel Inafuku, Chi Xue, Luyu Gao, Ze Yang, Yaïr Hein, Yonatan Kahn, Kevin Zhou, Di Luo, John Drew Wilson, Jarrod T. Reilly, Dmytro Bandak, Ofir Press, Liang Yang, Xueying Wang, Hao Tong, Nicolas Chia, Eliu Huerta, Hao Peng

TL;DR

CritPt establishes a physics reasoning benchmark composed of 71 unpublished, research-level challenges and 190 checkpoints across diverse frontier domains, designed and hand-curated by 50+ physicists. It employs a leakage-resistant, open-ended format and a physics-informed auto-grading pipeline to quantify end-to-end reasoning and modular task performance. Empirical results show that state-of-the-art LLMs exhibit only modest improvements with tooling and fail to solve full research-scale problems, underscoring a gap between AI capabilities and authentic physics research needs. The framework, including an interactive visualization platform, provides a concrete pathway for developing reliable AI assistants that can augment scientific discovery while keeping expert validation central.

Abstract

While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools.

Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

TL;DR

CritPt establishes a physics reasoning benchmark composed of 71 unpublished, research-level challenges and 190 checkpoints across diverse frontier domains, designed and hand-curated by 50+ physicists. It employs a leakage-resistant, open-ended format and a physics-informed auto-grading pipeline to quantify end-to-end reasoning and modular task performance. Empirical results show that state-of-the-art LLMs exhibit only modest improvements with tooling and fail to solve full research-scale problems, underscoring a gap between AI capabilities and authentic physics research needs. The framework, including an interactive visualization platform, provides a concrete pathway for developing reliable AI assistants that can augment scientific discovery while keeping expert validation central.

Abstract

While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason effectively through complex, open-ended challenges found in frontier physics research? And crucially, what kinds of reasoning tasks do physicists want LLMs to assist with? To address these questions, we present the CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point"), the first benchmark designed to test LLMs on unpublished, research-level reasoning tasks that broadly covers modern physics research areas, including condensed matter, quantum physics, atomic, molecular & optical physics, astrophysics, high energy physics, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics. CritPt consists of 71 composite research challenges designed to simulate full-scale research projects at the entry level, which are also decomposed to 190 simpler checkpoint tasks for more fine-grained insights. All problems are newly created by 50+ active physics researchers based on their own research. Every problem is hand-curated to admit a guess-resistant and machine-verifiable answer and is evaluated by an automated grading pipeline heavily customized for advanced physics-specific output formats. We find that while current state-of-the-art LLMs show early promise on isolated checkpoints, they remain far from being able to reliably solve full research-scale challenges: the best average accuracy among base models is only 5.7%, achieved by GPT-5 (high), moderately rising to around 10% when equipped with coding tools. Through the realistic yet standardized evaluation offered by CritPt, we highlight a large disconnect between current model capabilities and realistic physics research demands, offering a foundation to guide the development of scientifically grounded AI tools.

Paper Structure

This paper contains 23 sections, 2 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Design choices of CritPt
  3. Source and coverage: hand-curated research challenges from the physics community
  4. Benchmark criteria: leakage-resistant and reasoning-focused design
  5. Quality control: iterative development and multi-level expert review
  6. Structure of a challenge: an example
  7. Evaluation pipeline
  8. Two-step answer generation from models
  9. Auto-grading system
  10. Results
  11. Challenge-level evaluation: can LLMs solve unseen research problems?
  12. Checkpoint-level evaluation: smaller tasks that LLMs can assist today?
  13. Reliability metric: can we trust LLM outputs?
  14. Detailed analysis of full model responses
  15. Conclusion
  16. ...and 8 more sections

Figures (6)

  • Figure 1: CritPt's challenges (left) and checkpoints (right) cover three flavors of physics research -- theoretical, experimental, and computational -- encountered by physics researchers.
  • Figure 2: A schematic overview of the two-step generation process and the grading system. Left: The two-step generation protocol separates problem-solving (first round) from answer formatting (second round). Right: The automated grading system compares the model output against the gold answer from experts using scripts customized according to the expected answer format.
  • Figure 3: A comparison of 10 models' performance on 70 test CritPt challenges. Each model is tested on every challenge in five independent runs. Main plot: the average accuracy over all runs and all challenges for each model. Inset a: the average number of reasoning tokens used per run for each model. Inset b: the average cost (USD) per run, calculated from token usage and API pricing for each model (\ref{['SI:API_stat']}).
  • Figure 4: Schematic of the two experimental setups for evaluating sequential checkpoints within a multi-turn conversation. (a) Self-carryover without expert answer: The model's own output from the previous checkpoint is used as context for the next one. (b) Oracle carryover with expert answers: The correct answer (shown in red) to the previous checkpoint is provided before the model attempts the next checkpoint.
  • Figure 5: A comparison of 10 models' performance on the 187 test CritPt checkpoints. The average accuracy is aggregated over all runs and all checkpoints, in two setups respectively. Solid bar reports the self-carryover without expert answer, while the hatched pattern reports oracle carryover with expert answers.
  • ...and 1 more figures