A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment

TL;DR

This work introduces LLM-AAI, a framework for evaluating the physical common-sense reasoning of large language models by embedding them as agents within the Animal-AI 3D environment, enabling ecologically valid and directly comparable assessments against reinforcement learning agents and humans. By leveraging the Animal-AI Testbed Olympics and two prompting strategies, the authors show that state-of-the-art multimodal LLMs can solve a subset of simple tasks but generally underperform human children and top RL entrants on more complex tasks. The study demonstrates the feasibility and value of embodied evaluation for LLMs, discusses current limitations such as observation cadence and coarse control, and outlines concrete avenues for improvement including frame-by-frame control and memory-augmented planning. Overall, the paper argues that embodied, cognitively grounded testing enhances the predictability and reliability of LLMs in real-world physical reasoning and enables meaningful cross-species comparisons.

Abstract

As general-purpose tools, Large Language Models (LLMs) must often reason about everyday physical environments. In a question-and-answer capacity, understanding the interactions of physical objects may be necessary to give appropriate responses. Moreover, LLMs are increasingly used as reasoning engines in agentic systems, designing and controlling their action sequences. The vast majority of research has tackled this issue using static benchmarks, comprised of text or image-based questions about the physical world. However, these benchmarks do not capture the complexity and nuance of real-life physical processes. Here we advocate for a second, relatively unexplored, approach: 'embodying' the LLMs by granting them control of an agent within a 3D environment. We present the first embodied and cognitively meaningful evaluation of physical common-sense reasoning in LLMs. Our framework allows direct comparison of LLMs with other embodied agents, such as those based on Deep Reinforcement Learning, and human and non-human animals. We employ the Animal-AI (AAI) environment, a simulated 3D virtual laboratory, to study physical common-sense reasoning in LLMs. For this, we use the AAI Testbed, a suite of experiments that replicate laboratory studies with non-human animals, to study physical reasoning capabilities including distance estimation, tracking out-of-sight objects, and tool use. We demonstrate that state-of-the-art multi-modal models with no finetuning can complete this style of task, allowing meaningful comparison to the entrants of the 2019 Animal-AI Olympics competition and to human children. Our results show that LLMs are currently outperformed by human children on these tasks. We argue that this approach allows the study of physical reasoning using ecologically valid experiments drawn directly from cognitive science, improving the predictability and reliability of LLMs.

Figures (48)

• Figure 1: One task from each of the ten levels of the Animal-AI Testbed. The aim in every task is to collect as many yellow and/or green spheres while avoiding red zones, orange zones, and red spheres, before time runs out. Blue arrows indicate the location of the agent, and green arrows indicate the location of green spheres. The rightmost images show the agent's perspective during play in levels 5 and 10.
• Figure 2: LLM-AAI. LLMs generate actions such as Turn(45); and pass them to LLM-AAI. LLM-AAI then parses these actions into commands that are understandable to the AAI environment and where they are subsequently executed. Observations from the environment are passed back to LLM-AAI, which concatenates them into the observation history and provides them, along with prompts like "Your remaining health is 80.6", to the LLM for reasoning and planning its next actions.
• Figure 3: The proportion of trials passed by each LLM on each level, consisting of 3 trials of 4 tasks each (total n=12 trials per level). The interquartile range of proportions for all children (n=59) and the top 10 entrants to the Animal-AI Olympics Competition are presented as bars, with overall proportion for those populations indicated by points. Note that the children and competition agents have error bars, while the LLMs do not. This is because the child and competition agents contain a population of different individuals, across which we would like to understand variation, while the LLMs are repetitions of the same individual, and so are aggregated into a single value.
• Figure 4: The proportion of trials by each LLM on each level, consisting of 3 trials of 4 tasks each (total n=12 trials per level). The interquartile range of proportions for all children (n = 59) and the top 10 entrants to the Animal-AI Olympics Competition are presented as bars, with overall proportion for those populations indicated by points.
• Figure 5: $\langle$ Initial image: no response $\rangle$