Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Formalizing Embeddedness Failures in Universal Artificial Intelligence

Cole Wyeth, Marcus Hutter

TL;DR

The paper formalizes embeddedness concerns for the AIXI agent by modeling joint action-percept sequences with a universal mixture and introducing a joint variant, JAIXI. It develops the env and dual mappings to connect sequence distributions with environments, proving dominance relations and analyzing both positive results (normalization enabling learning in deterministic adversarial environments) and negative results (adversarial non-convergence in simple cases). The key contributions include the definition of JAIXI, analysis of its learning dynamics, and insights into how normalization and reflective variants can address embeddedness challenges. This work clarifies fundamental limits and potential remedies for embedded agency in universal AI, informing theoretical foundations and future AGI design choices.

Abstract

We rigorously discuss the commonly asserted failures of the AIXI reinforcement learning agent as a model of embedded agency. We attempt to formalize these failure modes and prove that they occur within the framework of universal artificial intelligence, focusing on a variant of AIXI that models the joint action/percept history as drawn from the universal distribution. We also evaluate the progress that has been made towards a successful theory of embedded agency based on variants of the AIXI agent.

Formalizing Embeddedness Failures in Universal Artificial Intelligence

TL;DR

The paper formalizes embeddedness concerns for the AIXI agent by modeling joint action-percept sequences with a universal mixture and introducing a joint variant, JAIXI. It develops the env and dual mappings to connect sequence distributions with environments, proving dominance relations and analyzing both positive results (normalization enabling learning in deterministic adversarial environments) and negative results (adversarial non-convergence in simple cases). The key contributions include the definition of JAIXI, analysis of its learning dynamics, and insights into how normalization and reflective variants can address embeddedness challenges. This work clarifies fundamental limits and potential remedies for embedded agency in universal AI, informing theoretical foundations and future AGI design choices.

Abstract

We rigorously discuss the commonly asserted failures of the AIXI reinforcement learning agent as a model of embedded agency. We attempt to formalize these failure modes and prove that they occur within the framework of universal artificial intelligence, focusing on a variant of AIXI that models the joint action/percept history as drawn from the universal distribution. We also evaluate the progress that has been made towards a successful theory of embedded agency based on variants of the AIXI agent.

Paper Structure

This paper contains 7 sections, 5 theorems, 16 equations.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mathematical Preliminaries
  4. The Dualistic Mixture
  5. Joint AIXI
  6. Discussion
  7. List of Notation

Key Result

Theorem 6

There exists $\omega\in\mathbb{B}^\infty$ with $\omega_{2n} = \omega_{2n-1}$ but $\liminf_n \xi_U(\omega_{2n}|\omega_{1:2n-1}) < 1$.

Theorems & Definitions (11)

  • Definition 1: lower semicomputable
  • Definition 2: semimeasure
  • Definition 3: $\xi_U$
  • Definition 4: $\xi^\text{AI}$
  • Definition 5: domination
  • Theorem 6: Adversarial non-convergence of $\xi_U$
  • Theorem 7: Adversarial non-convergence of $\xi^U$
  • Theorem 8
  • Conjecture 9
  • Theorem 10: Adversarial learning for $\hat{\xi}_U$
  • ...and 1 more