Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models

TL;DR

This work tackles the knowledge gaps and misapplication failures of large language models on physics problems by introducing Physics Reasoner, a knowledge-augmented framework that combines a $122$-formula formula set with task-specific checklists and Python-based computation. The system operates in three stages—problem analysis, formula retrieval, and guided reasoning—guided by checklists to improve both knowledge acquisition and correct application, and it achieves a $5.8\%$ average accuracy improvement on the SciBench benchmark. Empirical results show state-of-the-art performance across multiple models and datasets, with ablation studies confirming the necessity of the formula set and checklists, as well as favorable token efficiency. The approach demonstrates the value of explicit physics knowledge and guided application for enhancing LLM-based physics problem solving and suggests that expanding the formula set could further improve capabilities.

Abstract

Physics problems constitute a significant aspect of reasoning, necessitating complicated reasoning ability and abundant physics knowledge. However, existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application. To mitigate these issues, we propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs. Specifically, the proposed framework constructs a comprehensive formula set to provide explicit physics knowledge and utilizes checklists containing detailed instructions to guide effective knowledge application. Namely, given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning. During the process, checklists are employed to enhance LLMs' self-improvement in the analysis and reasoning stages. Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.

Figures (10)

• Figure 1: Exemplar error cases. Given a complex physics problem, LLMs may make mistakes due to a lack of physics knowledge, as illustrated in (a), or incorrect application of knowledge, as shown in (b). The ground truth answer to this question is shown in (c).
• Figure 2: Distribution of the proposed three error types across Sys, CoT, and PoT baselines.
• Figure 3: Example for each error type, where the red highlighted parts indicate errors.
• Figure 4: Illustration of Physics Reasoner, solving a physics problem using LLMs with the help of the formula set and checklists. The approach contains three stages: problem analysis, knowledge retrieval, and guided reasoning.
• Figure 5: An example of our formula annotation.