Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Value Under Ignorance in Universal Artificial Intelligence

Cole Wyeth, Marcus Hutter

TL;DR

The paper broadens universal AI by allowing arbitrary continuous utilities beyond simple rewards and reframes semimeasure loss as either death or model-ignorance within an imprecise-probability framework. It develops a rigorous semimeasure-extension method to define integration and shows that Choquet-integral expected utilities align with traditional recursive value functions, while also enabling a credal-set interpretation that can improve computability. The authors prove existence of optimal policies under continuity assumptions and analyze computability properties, highlighting potential benefits of the imprecise-probability approach for general utility AIs. The work offers a foundation for decision-theoretic, history-based universal agents with richer goal structures and discusses implications for AI alignment and the semantics of termination.

Abstract

We generalize the AIXI reinforcement learning agent to admit a wider class of utility functions. Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the agent's belief distribution only predict a finite prefix of the history, which is sometimes interpreted as implying a chance of death equal to a quantity called the semimeasure loss. This death interpretation suggests one way to assign utilities to such history prefixes. We argue that it is as natural to view the belief distributions as imprecise probability distributions, with the semimeasure loss as total ignorance. This motivates us to consider the consequences of computing expected utilities with Choquet integrals from imprecise probability theory, including an investigation of their computability level. We recover the standard recursive value function as a special case. However, our most general expected utilities under the death interpretation cannot be characterized as such Choquet integrals.

Value Under Ignorance in Universal Artificial Intelligence

TL;DR

The paper broadens universal AI by allowing arbitrary continuous utilities beyond simple rewards and reframes semimeasure loss as either death or model-ignorance within an imprecise-probability framework. It develops a rigorous semimeasure-extension method to define integration and shows that Choquet-integral expected utilities align with traditional recursive value functions, while also enabling a credal-set interpretation that can improve computability. The authors prove existence of optimal policies under continuity assumptions and analyze computability properties, highlighting potential benefits of the imprecise-probability approach for general utility AIs. The work offers a foundation for decision-theoretic, history-based universal agents with richer goal structures and discusses implications for AI alignment and the semantics of termination.

Abstract

We generalize the AIXI reinforcement learning agent to admit a wider class of utility functions. Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the agent's belief distribution only predict a finite prefix of the history, which is sometimes interpreted as implying a chance of death equal to a quantity called the semimeasure loss. This death interpretation suggests one way to assign utilities to such history prefixes. We argue that it is as natural to view the belief distributions as imprecise probability distributions, with the semimeasure loss as total ignorance. This motivates us to consider the consequences of computing expected utilities with Choquet integrals from imprecise probability theory, including an investigation of their computability level. We recover the standard recursive value function as a special case. However, our most general expected utilities under the death interpretation cannot be characterized as such Choquet integrals.

Paper Structure

This paper contains 8 sections, 4 theorems, 17 equations.

Table of Contents

  1. Introduction
  2. Background
  3. Mathematical Preliminaries
  4. Semimeasure Extension
  5. Prediction with Semimeasures
  6. Value Functions and Integration
  7. AIXI with General Utility Function
  8. Discussion

Key Result

theorem 1

A probability pre-semimeasure $\nu_0$ on ${\cal A}^*$ defines a unique probability measure $P$ on $(\Omega',\mathcal{F}')$ satisfying defines an extension of $\nu_0$ to $(\Omega,\mathcal{F})$, with $P(x) = L_\nu(x)$.

Theorems & Definitions (13)

  • definition 1: lower semicomputable
  • definition 2: upper semicomputable
  • definition 3: estimable
  • definition 4: pre-semimeasure
  • definition 5: semimeasure
  • definition 6: semimeasure loss
  • theorem 1: Semimeasure extension
  • definition 7: Choquet integral $\nu$
  • theorem 2: $V^\pi_\mu = \sf C\!\!\!\!\!\!\int (\Sigma_t \gamma_t r_t) d\mu^\pi$
  • lemma 1: $\int \underline{f}dP_\nu = \sf C\!\!\!\!\!\!\int f d\nu$
  • ...and 3 more