States as goal-directed concepts: an epistemic approach to state-representation learning

TL;DR

This work argues that state representations should be construed as goal-directed concepts, formalizing telic states as equivalence classes $S_g = \Delta(\mathcal{H})/\sim_g$ over experience distributions. It develops a formal framework for inferring an agent's goals from behavior in a task, and applies it to an odor-guided rat experiment using a parameterized goal $g_\beta$ and the Goal Alignment Coefficient $GAC_\beta(h) = g_\beta(h)/n$. Empirically, optimized $\beta^*$ align animal histories with the proposed goals more than random baselines, supporting a view of state representations as arising from goals; the framework also yields informative telic-state trajectories. The supplementary material extends the theory to policy learning via KL-based information projections and introduces transition-sensitive goals, connecting goal-directed behavior to an information-cost over policies and exhibiting probability-matching through a principled objective.

Abstract

Our goals fundamentally shape how we experience the world. For example, when we are hungry, we tend to view objects in our environment according to whether or not they are edible (or tasty). Alternatively, when we are cold, we may view the very same objects according to their ability to produce heat. Computational theories of learning in cognitive systems, such as reinforcement learning, use the notion of "state-representation" to describe how agents decide which features of their environment are behaviorally-relevant and which can be ignored. However, these approaches typically assume "ground-truth" state representations that are known by the agent, and reward functions that need to be learned. Here we suggest an alternative approach in which state-representations are not assumed veridical, or even pre-defined, but rather emerge from the agent's goals through interaction with its environment. We illustrate this novel perspective by inferring the goals driving rat behavior in an odor-guided choice task and discuss its implications for developing, from first principles, an information-theoretic account of goal-directed state representation learning and behavior.

Figures (1)

• Figure 1: Optimized weight parameters for liquid amount (left), delay duration (center) and side choice (right) preferences. Optimized $\beta$ values maximizing the goal-alignment coefficient of empirical histories (orange) are significantly larger than those of simulated random actions yoked to the observation histories of each animal (purple) for big vs. small amount and short vs. long delay but not for right vs. left nose pokes. Solid lines show Gaussian kernel density distribution estimates. Asterisks indicate significance levels (paired t-test, $^*p<0.01$; $^{**}p<0.001$).