How Well Do AI Systems Solve AP Physics? A Comparative Evaluation of Large Language Models on Algebra-Based Free Response Questions

TL;DR

It is suggested that while contemporary AI systems can effectively support routine physics problem solving, they remain limited in tasks requiring spatial reasoning, visual interpretation, and conceptual integration.

Abstract

The rapid advancement of LLMs has generated growing interest in their potential role in physics education and assessment, yet a focused evaluation of their performance on multi-faceted, free-response physics problems remains underexplored. In this study, we systematically evaluate the performance of four widely accessible AI systems-ChatGPT 4.1 mini, Gemini 2.5 Flash, Claude 4.0 Sonnet, and DeepSeek R1-on AP Physics 1 and 2 free-response questions administered between 2015 and 2025. Model-generated solutions were produced under standardized exam-style prompting and evaluated by three independent physics experts using official College Board scoring guidelines. All models achieved relatively high mean scores (82-92%), indicating strong capability in structured algebraic problem solving. However, substantial year-to-year variability was observed, particularly for AP Physics 1, where statistical testing revealed no consistent performance hierarchy among models. In contrast, AP Physics 2 results showed statistically significant differences, with Gemini and DeepSeek demonstrating more consistent performance than Claude. A qualitative analysis revealed recurring error patterns across all models, including misinterpretation of diagrams and graphs, incorrect graph construction, incorrect reasoning about vector direction, circuit topology errors, partial and misleading qualitative explanations, and difficulties applying three-dimensional concepts such as the right-hand rule. These findings suggest that while contemporary AI systems can effectively support routine physics problem solving, they remain limited in tasks requiring spatial reasoning, visual interpretation, and conceptual integration. The results highlight both the instructional potential and current pedagogical limitations of AI-assisted learning tools in physics education.

Figures (3)

• Figure 1: Performance distributions combining violin plots (probability density) and box plots (statistical summaries). Violin shapes reveal distribution characteristics: wider regions indicate score concentrations while narrow regions show sparse data. Overlaid box plots display median (thick red line), interquartile range (box), whiskers (range), and outliers (red dots). (left) Physics 1 shows broader distributions with some bimodality, particularly for ChatGPT and Gemini, indicating inconsistent performance across exam years. (right) Physics 2 displays contrasting patterns: Gemini and DeepSeek exhibit narrow, sharply peaked distributions indicating consistent high performance, while ChatGPT shows greater spread with an outlier at 100% (2025 exam), illustrating capacity for perfect performance but lower typical scores.
• Figure 2: Temporal trends in AI model performance with inter-rater variability. Mean scores (solid lines with markers) and standard deviation bands (shaded regions) across three independent raters for (left) AP Physics 1 and (right) AP Physics 2. The shaded regions illustrate scoring consistency, with narrower bands indicating higher inter-rater agreement. Substantial year-to-year fluctuations are evident, with certain exam years (e.g., 2021) eliciting consistently high performance across all models while others (e.g., 2017, 2022) proved universally challenging. The crossing and diverging trajectories indicate that model rankings are not stable but rather depend on specific exam characteristics.
• Figure 3: Stability of model rankings across exam years, directly visualizing the Kendall's $W$ concordance findings. Each line traces a single model's rank (1 = highest performance, 4 = lowest) across the 10 exam years for (Left) AP Physics 1 and (Right) AP Physics 2. In Physics 1, frequent crossovers and rank reversals illustrate the low concordance ($W = 0.182$), with all models occupying each rank position multiple times and no consistent performance hierarchy emerging. Physics 2 shows more stable rankings ($W = 0.532$), with Gemini and DeepSeek predominantly occupying ranks 1-2, while ChatGPT more frequently ranks lower, consistent with the significant Friedman test and post-hoc findings.