Scaling Physical Reasoning with the PHYSICS Dataset

TL;DR

PHYSICS tackles the scarcity and evaluation biases in physics reasoning for LLMs by introducing a large-scale, bilingual physics dataset with broad domain and difficulty coverage, plus a dedicated Rule+Model evaluation framework to address unit conversion and numerical simplification challenges. It provides 14,568 training and 2,000 test samples (16,568 after translation) across Mechanics, Electromagnetism, Thermodynamics, Optics, and Modern Physics, with reasoning paths for training. The authors demonstrate that current models struggle with physics tasks, but fine-tuning on PHYSICS improves performance on physics benchmarks and enables transfer to mathematics, while the Rule+Model evaluator yields more accurate judgments than rule-based or model-based methods alone. They also present a rigorous data-collection and leakage-detection pipeline, strict quality control, and comprehensive error analysis that highlights knowledge deficits as a primary hurdle, guiding future domain-specific improvements and multi-modal extensions.

Abstract

Large Language Models (LLMs) have achieved remarkable progress on advanced reasoning tasks such as mathematics and coding competitions. Meanwhile, physics, despite being both reasoning-intensive and essential to real-world understanding, received limited academic and industrial attention. This paper introduces PHYSICS, a dataset containing 16,568 high-quality physics problems spanning subjects and difficulty levels, to facilitate this issue. Specifically, PHYSICS is curated with exercises from over 100 textbooks through a carefully designed pipeline for quality control. It covers five major physics domains: Mechanics, Electromagnetism, Thermodynamics, Optics, and Modern Physics. It also spans a wide range of difficulty levels, from high school to graduate-level physics courses. To utilize the data for improving and evaluating the model's physical reasoning capabilities, we split the dataset into training and test sets, and provide reasoning paths generated by powerful reasoning models for the training data to facilitate model training. In addition, for the evaluation part, we find that existing evaluation frameworks exhibit biases in aspects such as units, simplification, and precision in physics domain. To balance efficiency and accuracy, we introduce a Rule+Model evaluation framework tailored to physics problems. Our evaluations on current state-of-the-art open-source and proprietary models highlight the limitations of current models in handling physics-related tasks. We hope that our dataset and evaluation methodology will jointly advance the development of LLMs in the field of physics. The code and data can be found at: https://github.com/Zhengsh123/PHYSICS.

Figures (12)

• Figure 1: Pipeline of PHYSICS construction process (left) and characteristics of PHYSICS (right).
• Figure 2: Difficulty Distribution of PHYSICS.
• Figure 3: Answer types.
• Figure 4: Performance of the (partial) models across different physics subjects.
• Figure 5: Performance of the (partial) model across different physics difficulties.