PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors

TL;DR

PhysGym presents a novel benchmark and simulation platform to rigorously assess LLM-driven scientific reasoning in interactive physics, with fine-grained control over prior knowledge to dissect deductive and exploratory capabilities. The framework leverages 97 PHYBench-derived problems, a standardized simulation interface, and diverse evaluation metrics (including $R^2$, MSE, Kendall's $\tau$, and MAPE) to measure hypothesis fidelity and data fit under constrained experimental budgets. Key contributions include detailed prior-level design (L1–L4), robust evaluation protocols, and experimental results showing that prior information can both facilitate and constrain discovery, with non-monotonic patterns across problems. The findings highlight the need for improved exploration strategies and causal reasoning in AI scientists, and position PhysGym as a valuable tool for guiding the development of robust, domain-aware scientific agents.

Abstract

Evaluating the scientific discovery capabilities of large language model based agents, particularly how they cope with varying environmental complexity and utilize prior knowledge, requires specialized benchmarks currently lacking in the landscape. To address this gap, we introduce \textsc{PhysGym}, a novel benchmark suite and simulation platform for rigorously assessing LLM-based scientific reasoning in interactive physics environments. \textsc{PhysGym}'s primary contribution lies in its sophisticated control over the level of prior knowledge provided to the agent. This allows researchers to dissect agent performance along axes including the complexity of the problem and the prior knowledge levels. The benchmark comprises a suite of interactive simulations, where agents must actively probe environments, gather data sequentially under constraints and formulate hypotheses about underlying physical laws. \textsc{PhysGym} provides standardized evaluation protocols and metrics for assessing hypothesis accuracy and model fidelity. We demonstrate the benchmark's utility by presenting results from baseline LLMs, showcasing its ability to differentiate capabilities based on varying priors and task complexity.

Figures (10)

• Figure 1: Controlled levels of prior knowledge. PhysGym lists three types of prior knowledge: Context - a textual description of the environment, Variable Descriptions and Variable Names. The levels of prior start with full information disclosed to the model at Level 1 and then gradually strip the information from the model - removing the Context at Level 2, Variable Descriptions at Level 3 and then finally removing Variable Names as well by anonymizing variables at Level 4.
• Figure 2: Overview of the PhysGym suite.
• Figure 3: Success rates of different models by prior level.
• Figure 4: Model success rate as a function of prior knowledge for tasks grouped by dimensionality.
• Figure 5: The number of unique hypotheses proposed at different prior levels.