WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-making

TL;DR

WorldLLM tackles grounded world modeling for LLMs by injecting natural-language hypotheses into a forward-model prompt and iteratively refining these hypotheses through Bayesian inference, while driving informative data collection with curiosity-driven RL. The framework couples a Statistician (LLM-based forward model), a Scientist (hypothesis generator via Metropolis-like Bayesian updates), and an Experimenter (oracle or RL-driven data collector) to autonomously improve predictions without gradient-based LLM fine-tuning. In Playground-Text experiments, WorldLLM improves predictive accuracy and yields interpretable theories of environment dynamics, outperforming vanilla prompting and providing a principled alternative to fine-tuning. The study demonstrates the potential of combining in-context learning, theory induction, and intrinsic motivation to ground LLMs in domain-specific dynamics, while revealing challenges in generalization that guide future improvements in priors, hypothesis abstraction, and scalable architectures.

Abstract

Large Language Models (LLMs) possess general world knowledge but often struggle to generate precise predictions in structured, domain-specific contexts such as simulations. These limitations arise from their inability to ground their broad, unstructured understanding in specific environments. To address this, we present WorldLLM, a framework that enhances LLM-based world modeling by combining Bayesian inference and autonomous active exploration with reinforcement learning. WorldLLM leverages the in-context learning abilities of LLMs to guide an LLM-based world model's predictions using natural language hypotheses given in its prompt. These hypotheses are iteratively refined through a Bayesian inference framework that leverages a second LLM as the proposal distribution given collected evidence. This evidence is collected using a curiosity-driven reinforcement learning policy that explores the environment to find transitions with a low log-likelihood under our LLM-based predictive model using the current hypotheses. By alternating between refining hypotheses and collecting new evidence, our framework autonomously drives continual improvement of the predictions. Our experiments demonstrate the effectiveness of WorldLLM in a textual game environment that requires agents to manipulate and combine objects. The framework not only enhances predictive accuracy, but also generates human-interpretable theories of environment dynamics.

Figures (16)

• Figure 1: Our framework aims at improving an LLM's ability to predict the outcome of state-action pairs by augmenting it with natural language hypotheses about the world (Statistician). WorldLLM alternates between generating hypotheses (Scientist) and collecting evidence to update these hypotheses (Experimenter). The former uses Bayesian inference with an LLM as the proposal distribution. The latter uses curiosity-driven RL to train an agent to collect experiences that have a low likelihood under the Statistician with the current set of hypotheses.
• Figure 2: Our experimental setup. We use the Playground-Text environment that features objects the agent can interact with, along with textual observations and actions (a). We focus on 4 object types from Playground-Text: Water, Plants, Small Herbivores, and Big Herbivores. We show in (b) the technology tree for combining objects.
• Figure 3: Evolution of log-likelihoods computed by the Statistician (using the Scientist's last accepted hypotheses) on the test set throughout WorldLLM's iterations when using different oracles as Experimenters. We average per transition type over $8$ random seeds with the standard deviation represented by the error bars. We show at iteration 0 the log-likelihood from the Statistician without any hypotheses given.
• Figure 4: Proportion of collected transitions for RL-ALPEXP (left), RL-LogP (middle), and RL-ALP (right). The colored dashed lines correspond to the amount of transitions collected by O-Hardest.
• Figure 5: Log-likelihoods (averaged by transition type) produced by the Statistician when facing an instance of a Grow Plant transition. We show how using the generalization environment from Section \ref{['sec:worldllm-generalization']} affects the log-likelihoods (b).