General Agents Contain World Models, even under Partial Observability and Stochasticity

TL;DR

The paper addresses whether an agent inherently contains a usable model of its environment and extends prior results to show that even stochastic policies in partially observable, finite, communicating cMDPs must learn a world representation. It develops a goal-based framework and proves that transition probabilities $P(s'|s,a)$ can be estimated from the agent's behavior, with quantifiable error bounds that converge as the horizon depth grows. Key contributions include extending the theory to $(2n+1)$-th-degree $\delta$-optimal stochastic policies with rate $O(1/\sqrt{n})$, extending to partially observable settings, and showing that width-2 goals yield faster, more scalable estimation with rate $O(\log n / n)$. Together, these results establish that learning a world model is a fundamental consequence of capable behavior, even under randomness and partial observability, and they introduce practical reductions in goal complexity to make such learning tractable.

Abstract

Deciding whether an agent possesses a model of its surrounding world is a fundamental step toward understanding its capabilities and limitations. In [10], it was shown that, within a particular framework, every almost optimal and general agent necessarily contains sufficient knowledge of its environment to allow an approximate reconstruction of it by querying the agent as a black box. This result relied on the assumptions that the agent is deterministic and that the environment is fully observable. In this work, we remove both assumptions by extending the theorem to stochastic agents operating in partially observable environments. Fundamentally, this shows that stochastic agents cannot avoid learning their environment through the usage of randomization. We also strengthen the result by weakening the notion of generality, proving that less powerful agents already contain a model of the world in which they operate.

Figures (4)

• Figure 1: Example of a cMDP with states $\mathcal{S} = \{s_{-2}, s_{-1}, s_0, s_1, s_2\}$ and $\mathcal{A} = \{L, R\}$ where $p_R,p_L \in (0,1)$ are arbitrary probabilities.
• Figure 2: Diagram representing conceptually the policy $\pi^\star$ described in the proof of Lemma \ref{['lemma:prob_of_rho_depends_on_p']} with $p=P(s'|s,a)$. This figure was taken from richens2025general.
• Figure 3: Diagram of the proof of Theorem \ref{['teo:non-deterministic-policies-induce-transitions']} when $p=0.35$, $n=20$ and $\delta = 0.2$. The orange and blue lines (squares and circles) are the probabilities $P(X > k)$ and $P(X \leq k)$ for $k =0\ldots n$. The green and red lines (triangles and diamonds) represent feasible values of $p_{b,k}$ and $p_{a,k}$ for the different values of $k$ (they were chosen randomly). Note that our predictor $x$ would correspond to the vertical line at 6. In this case $\varepsilon = 0.125$, which means that for $k$ on the left of the black line at 5.5 we are guaranteed that $p_{b,k} \geq p_{a,k}$, and similarly, for $k \geq 8$ we know that $p_{a,k} \geq p_{b,k}$. Inside the black bars both options are possible, and thus the predictor $x$ could have turned out to be $5$ or $7$ instead.
• Figure 4: A partially observable cMDP with three states and two actions. An edge $s \overset{a}{\rightarrow} s'$ indicates that $P(s'|s,a) = 1$. Both states $s_1$ and $s_2$ are related to a unique observation $o_1$ (i.e. $\Omega(o_1|s_1) = \Omega(o_2,s_2) = 1$). State $s_3$ has its own observation $o_2$.

Theorems & Definitions (26)

• Definition 1
• Example 1
• Definition 2
• Example 2
• Example 3
• Example 4
• Lemma 1
• proof
• Definition 3
• Example 5