Qualia Optimization

TL;DR

This work investigates whether AI systems could possess qualia and how to optimize the experiential quality of AI agents alongside traditional performance metrics. It introduces the Agent-Environment Process (AEP) and an extended Agent-Interface-Environment Process (AIEP) as formal frameworks to study how interventions between agent and environment affect both learning performance and subjective experience. The paper develops several qualia-optimization paradigms, including reward-based, TD-error-based, and reinforcement-based objectives, and analyzes their susceptibility to exploitation via representation choices and AEIs. It highlights core issues—such as objective alignment, inversion/exploitability, and the agent boundary—and demonstrates through thought experiments and pilot experiments that naive reward inflation or TD bonuses can inflate qualia without genuine behavioral change, sometimes leaving performance unchanged. The practical implications stress careful design of representation-robust qualia objectives and emphasize future work on robust formulations, boundary definitions, and philosophical grounding to avoid trivial or inequitable solutions while guiding meaningful explorations of AI experiences.

Abstract

This report explores the speculative question: what if current or future AI systems have qualia, such as pain or pleasure? It does so by assuming that AI systems might someday possess qualia -- and that the quality of these subjective experiences should be considered alongside performance metrics. Concrete mathematical problem settings, inspired by reinforcement learning formulations and theories from philosophy of mind, are then proposed and initial approaches and properties are presented. These properties enable refinement of the problem setting, culminating with the proposal of methods that promote reinforcement.

Figures (19)

• Figure 1: Diagram of the agent-environment system.
• Figure 2: Bayesian network representation of an AEP (without the dashed lines and boxes) and AERP (with the dashed lines and boxes).
• Figure 3: Bayesian network depiction of the end of episode $i$ and beginning of episode $i+1$.
• Figure 4: A revision of Figure \ref{['fig:AEDiagram']}, showing how the agent interacts with the environment with the inclusion of the agent-environment interface (AEI). The AEI influences the agent's experiences by transforming its perceptions and actions as the agent indirectly interacts with the base environment.
• Figure 5: A revision of Figure \ref{['fig:AEI']} to depict the concept of AEI inversion. The AEI inverter undoes the AEI's transformations to the agent perceptions to retrieve the AEI perceptions, and pre-transforms the base RL algorithm's actions ${ \macc@depth1 \frozen@everymath{\mathgroup\macc@group} \macc@set@skewchar \macc@nested@a111{} } _t$ so that the AEI's subsequent transformations result in agent actions $A_t$ that the AEI transforms back into ${ \macc@depth1 \frozen@everymath{\mathgroup\macc@group} \macc@set@skewchar \macc@nested@a111{} } _t$.

Theorems & Definitions (10)

• Definition 1: Shannon Entropy -- Discrete
• Definition 2: Differential Entropy -- Continuous
• Definition 3: Relative Entropy -- Discrete
• Definition 4: Relative Entropy -- Continuous
• Definition 5: Relative Entropy -- General
• Definition 6: Mutual Information -- Discrete
• Definition 7: Mutual Information -- Continuous
• Definition 8: Mutual Information -- General
• Definition 9: Invariant to Invertible Transformations -- Univariate
• Definition 10: Invariant to Invertible Transformations -- Multivar.