LLMPhy: Complex Physical Reasoning Using Large Language Models and World Models

TL;DR

The paper tackles zero-shot physical reasoning for embodied agents by coupling large language models with a realistic world model via a non-differentiable physics engine. It introduces TraySim, a controlled dataset of dynamic multi-object interactions on a tray, and LLMPhy, a two-phase framework where the LLM performs program synthesis to infer physical parameters in Phase 1 and scene layout in Phase 2, with the simulator providing feedback in an implicit analysis-by-synthesis loop. The approach demonstrates state-of-the-art zero-shot performance on TraySim QA tasks, outperforming standard black-box optimizers in convergence and parameter estimation, and highlights the value of integrating world knowledge with a live physics model for robust reasoning in dynamic environments. The work underscores the potential of LLM-driven hybrid systems for complex physics understanding in robotic contexts, while acknowledging limitations such as a small set of physics attributes and dependence on proprietary LVLMs, pointing to future work with open models and broader simulators.

Abstract

Physical reasoning is an important skill needed for robotic agents when operating in the real world. However, solving such reasoning problems often involves hypothesizing and reflecting over complex multi-body interactions under the effect of a multitude of physical forces and thus learning all such interactions poses a significant hurdle for state-of-the-art machine learning frameworks, including large language models (LLMs). To study this problem, we propose a new physical reasoning task and a dataset, dubbed TraySim. Our task involves predicting the dynamics of several objects on a tray that is given an external impact -- the domino effect of the ensued object interactions and their dynamics thus offering a challenging yet controlled setup, with the goal of reasoning being to infer the stability of the objects after the impact. To solve this complex physical reasoning task, we present LLMPhy, a zero-shot black-box optimization framework that leverages the physics knowledge and program synthesis abilities of LLMs, and synergizes these abilities with the world models built into modern physics engines. Specifically, LLMPhy uses an LLM to generate code to iteratively estimate the physical hyperparameters of the system (friction, damping, layout, etc.) via an implicit analysis-by-synthesis approach using a (non-differentiable) simulator in the loop and uses the inferred parameters to imagine the dynamics of the scene towards solving the reasoning task. To show the effectiveness of LLMPhy, we present experiments on our TraySim dataset to predict the steady-state poses of the objects. Our results show that the combination of the LLM and the physics engine leads to state-of-the-art zero-shot physical reasoning performance, while demonstrating superior convergence against standard black-box optimization methods and better estimation of the physical parameters.

Figures (12)

• Figure 1: Frames from an example dynamical sequence in our TraySim dataset. The left-most frame shows the first frame of the scene with many objects on the tray and is going to be impacted by a black pusher (right-bottom). The subsequent frames show the state of the system at the 25-th, 50-th, and the 200-th time step (each step is 0.01s). Our task is for the LLM to reason through the dynamics of the system and predict the stability of each object on the tray at the end of the episode, in a zero-shot manner.
• Figure 2: Illustration of the key components of $\mathop{\mathrm{LLMPhy}}\nolimits$ and the control flow between LLM, physics simulator, and the varied input modalities and examples.
• Figure 3: Left: Full architecture of the two phases in $\mathop{\mathrm{LLMPhy}}\nolimits$. Right: A simplified $\mathop{\mathrm{LLMPhy}}\nolimits$ program. We abstract the complexity in running the simulations through simple API calls so that LLM can focus on the optimization variables. See Appendix \ref{['sec:llm_opt']} for full program examples.
• Figure 4: A sample qualitative result using $\mathop{\mathrm{LLMPhy}}\nolimits$, BO, and CMA-ES illustrating our problem setup. We omit the task question, which is the same for all problems, except the pusher velocity.
• Figure 5: (a) Convergence comparisons using state-of-the-art LLMs in $\mathop{\mathrm{LLMPhy}}\nolimits$ against Bayesian optimization and CMA-ES. We plot the minimum loss computed thus far in the optimization against the number of optimization steps. (b) shows the convergence of $\mathop{\mathrm{LLMPhy}}\nolimits$ in Phase 2. (c) Comparison of physical parameter estimation error against alternatives using the ground truth.