Evolution of Rewards for Food and Motor Action by Simulating Birth and Death

TL;DR

The results show that biologically reasonable positive rewards for food acquisition and negative rewards for motor action can evolve from randomly initialized ones, however, it is found that the rewards for motor action diverge into two modes: largely positive and slightly negative.

Abstract

The reward system is one of the fundamental drivers of animal behaviors and is critical for survival and reproduction. Despite its importance, the problem of how the reward system has evolved is underexplored. In this paper, we try to replicate the evolution of biologically plausible reward functions and investigate how environmental conditions affect evolved rewards' shape. For this purpose, we developed a population-based decentralized evolutionary simulation framework, where agents maintain their energy level to live longer and produce more children. Each agent inherits its reward function from its parent subject to mutation and learns to get rewards via reinforcement learning throughout its lifetime. Our results show that biologically reasonable positive rewards for food acquisition and negative rewards for motor action can evolve from randomly initialized ones. However, we also find that the rewards for motor action diverge into two modes: largely positive and slightly negative. The emergence of positive motor action rewards is surprising because it can make agents too active and inefficient in foraging. In environments with poor and poisonous foods, the evolution of rewards for less important foods tends to be unstable, while rewards for normal foods are still stable. These results demonstrate the usefulness of our simulation environment and energy-dependent birth and death model for further studies of the origin of reward systems.

Figures (15)

• Figure 1: A schematic figure on our evolutionary framework. Agents learn to maximize the reward functions inherited from their parents. They reproduce children with mutated reward functions and die from aging or hunger.
• Figure 2: Upper: Hazard function $h(t)$ used in our experiments. Lower: Survival function $S(t)$ corresponding to the hazard function above.
• Figure 3: Birth function $b(e)$ used in our experiments.
• Figure 4: Simulation environment used in our experiments. Blue circles are agents, red circles are foods, and outer gray lines are walls. Thin gray lines around agents indicate distance sensors.
• Figure 5: Description of the environment. The left figure shows an agent, foods, distance sensors, and the positions of motor outputs. The right figure shows the ranges of an agent's mouth and each collision sensor.