Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning

Alexander Meulemans, Rajai Nasser, Maciej Wołczyk, Marissa A. Weis, Seijin Kobayashi, Blake Richards, Guillaume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento, Blaise Agüera y Arcas

TL;DR

This work places embedded Bayesian agents at the center of multi-agent learning, arguing that prospective learning requires agents to predict both percepts and their own actions within a world that includes other learning agents. By extending Solomonoff/AIXI foundations to embedded agency, it introduces the MUPI framework, which leverages grain-of-truth and structural similarities to enable consistent mutual prediction and infinite-order theory of mind. The authors develop novel equilibrium concepts (SEE/EE) and prove convergence results for embedded agents, including when planning horizons grow, and they introduce the Reflective Universal Inductor (RUI) and Reflective Oracles (RO) to solve recursive prediction challenges. The framework connects to foundation-model paradigms, informs active exploration, and offers philosophical insights into self-models, consciousness, and free will, while acknowledging practical limitations and directions for future work.

Abstract

The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the fact that those other agents are, in turn, forming beliefs about it to predict its future behavior, motivating agents to model themselves as part of the environment. Here, building upon foundational work on universal artificial intelligence (AIXI), we introduce a mathematical framework for prospective learning and embedded agency centered on self-prediction, where Bayesian RL agents predict both future perceptual inputs and their own actions, and must therefore resolve epistemic uncertainty about themselves as part of the universe they inhabit. We show that in multi-agent settings, self-prediction enables agents to reason about others running similar algorithms, leading to new game-theoretic solution concepts and novel forms of cooperation unattainable by classical decoupled agents. Moreover, we extend the theory of AIXI, and study universally intelligent embedded agents which start from a Solomonoff prior. We show that these idealized agents can form consistent mutual predictions and achieve infinite-order theory of mind, potentially setting a gold standard for embedded multi-agent learning.

Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning

TL;DR

This work places embedded Bayesian agents at the center of multi-agent learning, arguing that prospective learning requires agents to predict both percepts and their own actions within a world that includes other learning agents. By extending Solomonoff/AIXI foundations to embedded agency, it introduces the MUPI framework, which leverages grain-of-truth and structural similarities to enable consistent mutual prediction and infinite-order theory of mind. The authors develop novel equilibrium concepts (SEE/EE) and prove convergence results for embedded agents, including when planning horizons grow, and they introduce the Reflective Universal Inductor (RUI) and Reflective Oracles (RO) to solve recursive prediction challenges. The framework connects to foundation-model paradigms, informs active exploration, and offers philosophical insights into self-models, consciousness, and free will, while acknowledging practical limitations and directions for future work.

Abstract

The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the fact that those other agents are, in turn, forming beliefs about it to predict its future behavior, motivating agents to model themselves as part of the environment. Here, building upon foundational work on universal artificial intelligence (AIXI), we introduce a mathematical framework for prospective learning and embedded agency centered on self-prediction, where Bayesian RL agents predict both future perceptual inputs and their own actions, and must therefore resolve epistemic uncertainty about themselves as part of the universe they inhabit. We show that in multi-agent settings, self-prediction enables agents to reason about others running similar algorithms, leading to new game-theoretic solution concepts and novel forms of cooperation unattainable by classical decoupled agents. Moreover, we extend the theory of AIXI, and study universally intelligent embedded agents which start from a Solomonoff prior. We show that these idealized agents can form consistent mutual predictions and achieve infinite-order theory of mind, potentially setting a gold standard for embedded multi-agent learning.

Paper Structure

This paper contains 103 sections, 72 theorems, 452 equations, 3 figures, 17 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Mathematical Preliminaries
  4. Sequences
  5. Probability Background
  6. General Reinforcement Learning Setup
  7. General reinforcement learning.
  8. Multi-agent general reinforcement learning (MAGRL).
  9. Bayesian prediction and agents
  10. Bayesian mixture environment.
  11. Decoupled Bayes-optimal agent.
  12. Algorithmic probability, Solomonoff induction and AIXI
  13. Solomonoff induction.
  14. Universal artificial intelligence.
  15. Reflective oracles.
  16. ...and 88 more sections

Key Result

Theorem 2.4

For any policy $\pi$, consider a ground-truth environment-policy measure $\mu^\pi$ and a Bayesian mixture environment-policy measure $\xi^\pi.$ If $\xi^\pi \stackrel{\times}\geq \mu^\pi$,

Figures (3)

  • Figure 1.1: Illustration of embedded Bayesian agents.Top: In standard, decoupled approaches to Bayesian agents for RL, an agent considers their own policy as separate from the world they and others are making predictions about, maintaining beliefs about environments that might contain other agents, but not the ego-agent itself. Bottom: Embedded Bayesian agents, consider themselves to be embedded within a universe, the combination of the ego-agents policy with the environment which can contain other agents. As such, when making predictions they are not only predicting the environment percepts but also their own actions, allowing them to leverage structural similarities into their predictions and resulting behavior. This allows embedded Bayesian agents to converge to different equilibrium points in multi-agent settings.
  • Figure 3.1: Graphical models for decoupled Bayesian agents (left) and embedded Bayesian agents (right). We added deterministic nodes $h_{\leq t}$ (squares) that represent all the nodes $\{a_k, e_k\}_{k=1}^t$, to avoid clutter.
  • Figure 5.1: Comparison of Decoupled Agency (AIXI) and Embedded Agency (MUPI). For AIXI (left), the agent $\pi$ and the environment $\nu$ are decoupled programs, interacting via percepts ($e_t$) and actions ($a_t$) on separate tapes, with the percept tape acting as the output tape for the environment and the input tape for the agent, and vice versa for the action tape. In contrast, in MUPI (right), the agent and environment are unified into a joint universe $\lambda$, with access to a reflective universal inductor (RUI) $\rho$, generating interleaved actions $a_t$ and observations $e_t$ on a single output tape.

Theorems & Definitions (268)

  • Definition 2.1: Semimeasures and measures
  • Definition 2.2: Total variation distance
  • Definition 2.3: Dominance
  • Theorem 2.4: Convergence of $\xi$ to $\mu$ in total variation blackwell1962mergingHutter:24uaibook2
  • Remark 2.6: Prospective prediction
  • Definition 2.7: Computable
  • Definition 2.8: Lower semicomputable
  • Definition 2.9: Limit computable
  • Definition 2.10: Monotone Turing machine
  • Definition 3.1: A grain of uncertainty
  • ...and 258 more