Emergent World Beliefs: Exploring Transformers in Stochastic Games

TL;DR

This work explores whether transformer language models can learn internal world models in imperfect-information environments by training a GPT‑style model on Poker Hand History data and probing its activations. The authors demonstrate that the model develops both deterministic encodings (hand ranks) and stochastic encodings (equity) that correlate with belief‑state quantities in a Poker POMDP, with nonlinear probes decoding these structures. Activation visualizations reveal belief‑state–like geometry in the residual stream, supporting the view that the model maintains an internal representation of hidden cards and strategies. While promising, the study relies on synthetic PHH data and highlights limitations related to data diversity and generalization, pointing to future work in scaling, Bayesian belief extraction, and broader domain applicability.

Abstract

Transformer-based large language models (LLMs) have demonstrated strong reasoning abilities across diverse fields, from solving programming challenges to competing in strategy-intensive games such as chess. Prior work has shown that LLMs can develop emergent world models in games of perfect information, where internal representations correspond to latent states of the environment. In this paper, we extend this line of investigation to domains of incomplete information, focusing on poker as a canonical partially observable Markov decision process (POMDP). We pretrain a GPT-style model on Poker Hand History (PHH) data and probe its internal activations. Our results demonstrate that the model learns both deterministic structure, such as hand ranks, and stochastic features, such as equity, without explicit instruction. Furthermore, by using primarily nonlinear probes, we demonstrated that these representations are decodeable and correlate with theoretical belief states, suggesting that LLMs are learning their own representation of the stochastic environment of Texas Hold'em Poker.

Figures (9)

• Figure 1: MLP probe performance for hand-rank identification. Panels (a-c) show confusion matricies for Layer 0 using datasets balanced by limiting each hand-rank class to the 30th, 35th, and 40th percentile of its unique sample count. Darker diagonal cells indicate more accurate predictions. Representation of rarer hand ranks, such as two pairs, is improved with lower percentiles. Panel (d) shows probe accuracy across all layers using the 40th percentile dataset, with 95% confidence intervals across five random seeds. Together, results indicate that hand-rank information is encoded strongly and consistently. See Appendix \ref{['actionIdentification']} for additional deterministic experimental results.
• Figure 2: Probe performance on stochastic representations. Panel (a) shows predicted versus true hand equity for Layer 0, demonstrating that model activations contain information about winning probability despite incomplete information. Panel (b) shows the $R^2$ value of equity prediction across layers, showing that equity information is strongest in earlier layers of (0-5) and progressively diminishes deeper in the network, consistent with information-bottleneck–style compression.
• Figure 3: t-SNE visualized activation plots. Activations are clustered by hand-rank and conceptual similarity. Distinct clusters indicate that the model internally organizes hands according to rank or equity strength that follows. Multiple clusters for ranks such as “pair” suggest that the model learns more detailed sub-categories (e.g., types of pairs such as J and K) rather than treating each rank as a single class.
• Figure 4: Training Pipeline. We train for up to 20 epochs, with early stopping if validation accuracy declines for three consecutive epochs to prevent overfitting.
• Figure 5: Action identification performance using linear and MLP probes when the action token (e.g., f, cc, sm) is masked out during inference, to prevent models from "cheating". Panels (a) and (b) show confusion matrices for the linear probe and two-layer MLP probe respectively, evaluated on Layer 4 of the transformer. Both probes achieve similar accuracy ($\sim$80%), suggesting that the model’s internal activations already encode sufficient information about common actions and their situational context, but also show confusions between actions with similar local structure (e.g., cc vs. f).