One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration

TL;DR

OneLife tackles the problem of learning symbolic, executable world models for stochastic, high-variance environments from a single unguided episode. It represents environment dynamics as a probabilistic mixture of modular laws with preconditions, enabling targeted learning and scalable reasoning through a dynamic computation graph. The approach integrates an LLM-driven exploration policy, an atomic-law synthesizer, gradient-based law-parameter inference, and forward simulation to support planning entirely within the learned model. Evaluated on Crafter-OO, OneLife outperforms a strong baseline on multiple mechanics and demonstrates planning via imagined rollouts, highlighting its potential for autonomous discovery and reasoning about unknown environments.

Abstract

Symbolic world modeling requires inferring and representing an environment's transitional dynamics as an executable program. Prior work has focused on largely deterministic environments with abundant interaction data, simple mechanics, and human guidance. We address a more realistic and challenging setting, learning in a complex, stochastic environment where the agent has only "one life" to explore a hostile environment without human guidance. We introduce OneLife, a framework that models world dynamics through conditionally-activated programmatic laws within a probabilistic programming framework. Each law operates through a precondition-effect structure, activating in relevant world states. This creates a dynamic computation graph that routes inference and optimization only through relevant laws, avoiding scaling challenges when all laws contribute to predictions about a complex, hierarchical state, and enabling the learning of stochastic dynamics even with sparse rule activation. To evaluate our approach under these demanding constraints, we introduce a new evaluation protocol that measures (a) state ranking, the ability to distinguish plausible future states from implausible ones, and (b) state fidelity, the ability to generate future states that closely resemble reality. We develop and evaluate our framework on Crafter-OO, our reimplementation of the Crafter environment that exposes a structured, object-oriented symbolic state and a pure transition function that operates on that state alone. OneLife can successfully learn key environment dynamics from minimal, unguided interaction, outperforming a strong baseline on 16 out of 23 scenarios tested. We also test OneLife's planning ability, with simulated rollouts successfully identifying superior strategies. Our work establishes a foundation for autonomously constructing programmatic world models of unknown, complex environments.

Figures (6)

• Figure 1: OneLife synthesizes world laws from a single unguided (no environment-specific rewards / goals) episode in a hostile, stochastic environment. OneLife models the world as mixture of laws written in code with a precondition-effect structure, each governing an aspect of the world, and infers parameters for the mixture that best explain the observed dynamics of the world. The resulting world model (WM) provides a probability distribution over attributes of an object-oriented world state, such as the position of a particular zombie. OneLife outperforms a strong baseline in modeling $16/23$ core game mechanics tested, measured by MRR (Mean Reciprocal Rank) of the true next state (\ref{['sec:evaluation']}) under the WM's likelihood. See \ref{['box:zombie-chase-law']} for a synthesized zombie law.
• Figure 2: Illustration of the inference process. The active laws for each observable (defined by $\mathcal{I}_k(s_t, a)$) determine the structure of the computation graph, i.e., which laws and their corresponding parameters $\theta_i$ are related to which observables. This structure in turn informs the parameter updates. Shown here is a dataset with a single transition instance, in which the player (P) moves right; at the same time, a zombie (Z) independently moves left. this implicates two laws, PlayerMovementLaw and ZombieMovementLaw, while not implicating the InventoryUpdateLaw. As a result, the loss computation is only a function of $\theta_1$ and $\theta_2$. Note we use $Z$ here to denote the normalizing factor. Examples of synthesized laws can be seen in \ref{['sec:law-examples']}.
• Figure 3: Two evaluation metric categories described in \ref{['sec:evaluation']}. A world state of an environment usually has more than two keys (i.e. Crafter-OO's state (\ref{['appendix:worldstate-data-model']}) when populated has 100+ key-value pairs,) and often has nested values, but here we show a simplest case to explain the calculation of (normalized) edit distance. We create distractors for state ranking using mutators (\ref{['sec:mutators']}), which programatically modify the next state $s^\prime$ in a transition $(s, a, s^\prime)$ to be illegal under the true transition function. For example, one of our mutators allows a crafting action (e.g. making a stone pickaxe) to succeed even when the prequisites for the crafting are not met.
• Figure 4: Per-scenario state ranking performance of OneLife (Ours) versus PoE-World, measured by Mean Reciprocal Rank (MRR $\uparrow$). Scenarios are grouped by the core game mechanic they test. Horizontal lines show the average MRR across all scenarios in a group for OneLife and PoE-World. OneLife demonstrates a more accurate understanding of the environment's laws, achieving a higher average MRR and outperforming the baseline on the majority of individual scenarios.
• Figure 5: We show an example of plan execution withinOneLife's world model for the "Stone Miner" scenario. The task is to mine stone, and can only be successfully completed if a wooden pickaxe is obtained before attempting to mine stone. We simulate two plans within the world model. The effective plan carries out a multi-step sequence of gathering wood, crafting a wooden pickaxe, and then attempting to mine. The ineffective plan attempts to mine the stone directly. The world learned by OneLife correctly simulates causal game mechanics that cause the effective plan to succeed and the ineffective plan to fail. The frames are generated by rendering the structured states constructed by OneLife's learned transition function.