Active Inference as a Model of Agency

TL;DR

The paper questions reward maximisation as the sole basis of agency and proposes active inference as a canonical, physics-grounded alternative that unifies exploration and exploitation through minimising risk and ambiguity via the expected free energy $-\\\\log P(a_{>t} \\mid h_{\\le t})$. It derives agency from first principles, showing a decomposition into a KL term and an expected observation-entropy term, and then outlines a concrete algorithm (preferential inference, perceptual inference, planning as inference) that realizes this objective in POMDP-like settings. The approach yields a principled, information-aware, and risk-averse decision-making framework that can explain a wide range of behaviours and be rewritten to resemble traditional RL under explicit world-models. It also discusses scalability via hierarchical deep models and acknowledges limitations regarding representations, pointing to future work in learning and developmental aspects of the world model, with implications for neuroscience, robotics, and AI safety.

Abstract

Is there a canonical way to think of agency beyond reward maximisation? In this paper, we show that any type of behaviour complying with physically sound assumptions about how macroscopic biological agents interact with the world canonically integrates exploration and exploitation in the sense of minimising risk and ambiguity about states of the world. This description, known as active inference, refines the free energy principle, a popular descriptive framework for action and perception originating in neuroscience. Active inference provides a normative Bayesian framework to simulate and model agency that is widely used in behavioural neuroscience, reinforcement learning (RL) and robotics. The usefulness of active inference for RL is three-fold. \emph{a}) Active inference provides a principled solution to the exploration-exploitation dilemma that usefully simulates biological agency. \emph{b}) It provides an explainable recipe to simulate behaviour, whence behaviour follows as an explainable mixture of exploration and exploitation under a generative world model, and all differences in behaviour are explicit in differences in world model. \emph{c}) This framework is universal in the sense that it is theoretically possible to rewrite any RL algorithm conforming to the descriptive assumptions of active inference as an active inference algorithm. Thus, active inference can be used as a tool to uncover and compare the commitments and assumptions of more specific models of agency.

Figures (3)

• Figure 1: (A) This figure illustrates a human (agent process $h$) interacting with its environment (external process $s$), and the resulting partition into external $s$, observable $o$, and autonomous $a$ processes. The agent does not have direct access to the external process, but samples it through the observable process. The observable process constitutes the sensory epithelia (e.g., eyes and skin), which influences the environment through touch. The autonomous process constitutes the muscles and nervous system, which influences the sensory epithelia, e.g., by moving a limb, and the environment, e.g., through speech by activating vocal cords. Autonomous responses at time $t + \delta t$ may depend upon all the information available to the agent at time $t$, that is $h_{\leq t}$. Thus, the systems we are describing are typically non-Markovian. (B) An active inference agent is completely described by its prediction model $P(s,o \mid a)$ and its preference model $P(s,o)$. When the prediction model is a POMDP the preference model is a hidden Markov model. In this setting, the colour scheme illustrates the problem of agency at $t=1$: the agent must execute an action (in red) based on previous actions and observations (in grey), which are informative about external states and future observations (in white). When specifying an active inference agent, it is important that prediction and preference models coincide on those parts they have in common; in this example the likelihood map $P(o \mid s)$. Note that these models need not be Markovian.
• Figure 2: Preferential inference on POMDPs. This figure illustrates how one may infer preferences efficiently when the prediction model is a POMDP and the preference model is a hidden Markov model (c.f., Figure \ref{['fig: partition']}.B). The problem is illustrated at time $t=1$. We may approximate the posterior preferences over states and observations given available data $P(s,o \mid h_{\leq t})$ by the product of the predictions of past states and observations given available data $P(s_{\leq t},o_{\leq t} \mid h_{\leq t})$ and the preferences over future states and observations given the current state $P(s_{> t},o_{> t} \mid s_{t})$, c.f., \ref{['eq: pref inference factorisation']}. These two distributions can, in turn, be obtained from the prediction and preference models via approximate inference. More simply, when preferences $P(s_t,o_t)$ are i.i.d. for all $t$ (c.f., Figure \ref{['fig: T-Maze']}), the latter distribution simply equals future preferences $P(s_{> t},o_{> t})$.
• Figure 3: T-Maze environment. This Figure illustrates a simple sequential decision-making task with a temporal horizon of $2$. In other words, the agent must choose $a_1$ given $h_0=o_0$ and, subsequently, $a_2$ given $h_{\leq 1}= (a_1, o_{\leq 1})$. In more detail, $s_t$: The T-Maze has four possible spatial locations: middle, top-left, top-right, bottom. One of the top locations yields a reward of 1000$ (money bag), while the other yields a punishment of -1000$ (flying money bag)---per time-step spent in respective locations. The reward's location determines the context. The bottom arm contains a cue whose colour (blue or green) discloses the context. Together, location and context determine the external state. $o_t$: The agent always observes its spatial location. In addition, when it is at the top of the Maze, it receives the reward or the punishment; when it is at the bottom, it observes the colour of the cue. $a_t$: Each action corresponds to visiting one of the four spatial locations. $P(s_t)$: The agent prefers being at the reward's location ($-\log P(s_t) =1000$) and avoid the punishment's location ($-\log P(s_t) =-1000$). All other external states have a neutral preference ($-\log P(s_t)=0$). $s_0$: the agent starts in the middle location and the context is initialised at random. Together these datum determine prediction $P(s,o \mid a)$ and preference $P(s,o)$ models, which in turn, determine agency by minimisation of expected free energy (Section \ref{['sec: simulating agency']}).

Theorems & Definitions (4)

• Definition 2.1
• Remark 2.2
• Lemma D.1
• proof