Can Large Language Models Correctly Interpret Equations with Errors?

TL;DR

This paper proposes a two-step AlphaPhysics pipeline that uses open-source LLMs to extract and translate student-written equations into a standardized form for verification by SMT/CAS tools. It systematically evaluates multiple prompting strategies and model sizes on typed Physics Olympiad responses, including a neuro-symbolic feedback loop (LLM-Modulo and Consensus) that leverages Z3/SymPy for correctness checks. The main finding is that no open-source model achieved the target accuracy (~$96.5\%$ equivalence) needed for autonomous grading, with best results around $88\%$ accuracy and a notable computational cost trade-off. The work highlights practical implications for environmentally conscious, privacy-preserving grading and suggests decomposing tasks or exploring multi-modal approaches to improve performance, while emphasizing the need for reliable automated reasoning over algebraic formulas.

Abstract

This paper explores the potential of Large Language Models to accurately extract and translate equations from typed student responses into a standard format. This is a useful task as standardized equations can be graded reliably using a Computer Algebra System or a Satisfiability Modulo Theories solver. Therefore physics instructors interested in automated grading would not need to rely on the mathematical reasoning capabilities of Language Models. We used two novel frameworks to improve the translations. The first is consensus where a pair of models verify the correctness of the translations. The second is a neuro-symbolic LLM-modulo approach were models receive feedback from an automated reasoning tool. We performed experiments using responses to the Australian Physics Olympaid exam. We report on results, finding that no open-source model was able to translate the student responses at the desired level of accuracy. Future work could involve breaking the task into smaller components before parsing to improve performance, or generalizing the experiments to translate hand-written responses.

Figures (5)

• Figure 1: The AlphaPhysics Pipeline. The student response is parsed by an LLM. A symbolic reasoning engine then determines the student's grade.
• Figure 2: Plot of accuracy against model size. Colour indicates the technique being used. Circles correspond to thinking models (like Deepseek) while crosses refer to other models. Blue vertical lines indicate the maximum model size that fits in the GPU memory and the maximum model size which can be run on the hardware. Sigmoid functions show the overall trend of the data. In 3.5% of responses the graders could not agree on what the students intended to write. Therefore upper maroon horizontal line indicates the 96.5% accuracy where responses become ambiguous.
• Figure 3: Model Accuracy vs. Time to Run. Colour is used to indicate the technique. Circles correspond to thinking models (DeepSeek), all other models have crosses. Three horizontal maroon lines indicate the number of blank responses, the highest performance of any combination of models, and the highest expected performance as the final 3.5% of responses were determined by the markers to be ambiguous.
• Figure :
• Figure :