Exploration Through Introspection: A Self-Aware Reward Model

TL;DR

Problem: enable agents to model their own hidden mental states and study how self-awareness affects exploration and learning. Approach: embed a self-aware reward model that infers an internal pain state from happiness using a hidden Markov model and modulates learning in gridworlds via a subjective reward that subtracts a pain penalty. Key findings: introspective agents outperform baselines in stationary and non-stationary environments; chronic pain yields faster adaptation but can produce addiction-like relief-seeking and negative lifetime well-being; results illustrate plausible human-like dynamics and support a route toward unified Theory of Mind in AI. Significance: demonstrates how internal state inference can empower exploration and learning and points to future work extending introspection to infer other agents' states.

Abstract

Understanding how artificial agents model internal mental states is central to advancing Theory of Mind in AI. Evidence points to a unified system for self- and other-awareness. We explore this self-awareness by having reinforcement learning agents infer their own internal states in gridworld environments. Specifically, we introduce an introspective exploration component that is inspired by biological pain as a learning signal by utilizing a hidden Markov model to infer "pain-belief" from online observations. This signal is integrated into a subjective reward function to study how self-awareness affects the agent's learning abilities. Further, we use this computational framework to investigate the difference in performance between normal and chronic pain perception models. Results show that introspective agents in general significantly outperform standard baseline agents and can replicate complex human-like behaviors.

Figures (9)

• Figure 1: Non-stationary: Mean cumulative objective reward (COR) and standard deviation (SD) of the best performing agents from each reward function category across different self-awareness models. (*): Statistically significant improvement over the same category 'No pain' baseline (one-sided paired-samples t-test, $p \ll 0.05$). Sample size ${n=300}$.
• Figure 2: Normal pain: Parameters of the transition matrix ${\Pr(H_t \mid H_{t-1})}$, emission matrix ${\Pr(O_t \mid H_t)}$, and initial state distribution ${\Pr(H_0)}$ of the hidden Markov model for normal pain perception. Transitions favor recovery, and emissions distinguish noxious from harmless sensations. Adapted from Eckert2022Eckert2022.
• Figure 3: Chronic pain: Parameters of the transition matrix ${\Pr(H_t \mid H_{t-1})}$, emission matrix ${\Pr(O_t \mid H_t)}$, and initial state distribution ${\Pr(H_0)}$ of the hidden Markov model for chronic pain perception. Transitions are sticky and emissions are ambiguous. Adapted from Eckert2022Eckert2022.
• Figure 4: Basic environment setup. Yellow circle: agent. Green square: food state. Yellow-marked area: possible initial agent spawn positions. Green-marked area: possible initial food state spawn positions.
• Figure 5: Stationary environment: Mean cumulative objective reward (COR) and standard deviation (SD) of the best performing agents from each reward function category across different self-awareness models. (*): Statistically significant improvement over the same category 'No pain' baseline (one-sided paired-samples t-test, $p \ll 0.05$). Sample size $n=300$. To highlight the differences, the plot starts from 1000 on the y-axis.