How Intrinsic Motivation Underlies Embodied Open-Ended Behavior

Abstract

Although most theories posit that natural behavior can be explained as maximizing some form of extrinsic reward, often called utility, some behaviors appear to be reward independent. For instance, spontaneous motor babbling in human newborns and curiosity in little kids and other animals seem to elude a simple explanation in terms of extrinsic reward maximization. Rooted in these observations, intrinsic motivation has emerged as a potentially major driver of behavior. However, only recently have several quantitative and foundational theories of intrinsic motivation been put forward. We first provide a general framework to understand behavior as being organized hierarchically: objective--intrinsic reward, or motivation--drives, goals and extrinsic reward. We next review the main formalizations of intrinsic motivation, including empowerment, the free energy principle, information-gain maximization, and the maximum occupancy principle. These theories produce complex behavior by promoting, in various ways, entropic action-state paths. The presence of a single intrinsic motivation objective breaks infinite regress, as drives and goals act only temporarily to serve the objective. Extrinsic rewards, such as sugar or protein, are just a means to achieve the objective. Bounded cognition and embodiment impose constraints and boundary conditions for the intrinsic motivation objective. By virtue of their capability to generate complex behavior in a task-agnostic manner, theories of intrinsic motivation promise to become successful generative models of open-ended, embodied behavior.

Figures (3)

• Figure 1: Hierarchical organization of objectives, drives, goals and extrinsic rewards in three global theories of behavior. (a) Extrinsic reward maximization vonneumann2007theorykahneman2013prospectsutton1998reinforcement posits an extrinsic reward function (denoted as reward in the figure) as the objective of the behavior. Extrinsic reward signals can directly trigger behavior, such as eating, but often goal-directedness, involving planning, is needed to obtain distant or future extrinsic rewards. (b) Drive theory berridge2004motivation of behavior places drives and needs at the top of the hierarchy rosado2022drive. Drives compete between themselves to directly generate behavior or form temporary goals to satisfy them. Drives and needs are only satisfied if appropriate extrinsic rewards are gathered (e.g., water for thirst). (c) Intrinsic motivation theories jung2011empowermentramirez2024complexpathak2017curiosity put a single objective at the top of the hierarchy. The objective can be occupying action-state path space (Eq. \ref{['eq:objective']}). Generally, the objective consists in maximizing future cumulative intrinsic reward. This objective can generate drives and goals depending on the proximity of survival needs, which together generate behavior. Extrinsic rewards are obtained as a byproduct of behavior serving to the satisfaction of drives and needs, but they do not directly affect the objective. The intrinsic reward mostly depends on behavioral state transition probabilities and the agent's policy. Intrinsic and extrinsic rewards, needs and goals, and states and actions, are all internal to the agent, but they can represent the external reality, albeit in a potentially distorted and biased manner.
• Figure 2: Intrinsic motivation versus extrinsic reward maximization in locomotion pattern generation in an embodied agent. The intrinsically motivated ant (panel a) discovers diverse behaviors, including walking and jumping. This diversity arises because the agent is intrinsically motivated to occupy action-state path space (Eq. \ref{['eq:objective']}), pushing it to visit the full range of behaviors possible within its physical constraints. In contrast, the extrinsic reward maximizer (panel b) is driven solely by positive rewards proportional to rightward speed. Over time, its behavior simplifies, converging to a single strategy: running to the right (https://drive.google.com/file/d/1fqYDU8GRyppuSssb3vOhFVrlig_7hhAa/view?usp=drive_link). For both agents, a terminal state is reached when the ant’s torso touches the ground (indicating a fall). This terminal state introduces a natural boundary that limits and shapes behavior. The virtual agent used here is the Ant in the MuJoCo environment todorov2012mujocoramirez2024complex.
• Figure 3: Intrinsic motivation in the form of maximizing action-state path entropy (a) generates diverse stochastic behavior, (b) shows goal-directed behavior when food sources (green region) are available, (c) makes the agent avoid terminal states (black region), and (d) provides a formalism that satisfies the additive property. Two initial and two final action pairs are indicated, sampled from policy $\pi(a|s)$, where $s=(s_x,s_y)$ is a two dimensional state variable in this particular case for illustration; actions are selected throughout the paths (not shown). Cones indicate the physically accessible region of action-state path space. In (d), path occupancy from time $t=0$ to time $t=T$ can be computed as the occupancy from time $t=0$ to $t=T/2$, plus the occupancy from $t=T/2$ to $t=T$ averaged over the intermediate action-states at time $t=T/2$, which fulfills the additive property of action-state path entropy, Eq. \ref{['eq:entropy_additive']}. This allows global optimization to be divided into subproblems. Additivity also implies time-invariance (i.e., the future from $t=0$ looks the same as from $t=T/2$, everything else being the same), a fundamental law of nature.