Inclusive Fitness as a Key Step Towards More Advanced Social Behaviors in Multi-Agent Reinforcement Learning Settings

TL;DR

The paper tackles the challenge of fostering progressive, non-binary social behaviors in multi-agent reinforcement learning by importing inclusive fitness concepts from biology. It introduces a genotype-based inclusive reward where an agent's payoff is augmented by the weighted payoffs of genetically related agents, with weights given by a Hamming-based similarity $h(\mathbf{g}_i, \mathbf{g}_j)$ and a formal connection to Hamilton's rule $c < h b$. Through network PD experiments, it demonstrates that inclusive rewards promote cooperation in line with relatedness, and that population structure (opponent discrimination vs limited dispersal) significantly shapes emergent strategies. It then outlines an evolution-aware extension to Markov games, proposing three reward variants—longevity, replication, and combined rewards—to drive long-horizon strategic complexity and autocurriculum in open-ended environments like Neural MMO. If validated, this framework could yield socially intelligent agents capable of adapting cooperation and competition to dynamic social landscapes, bridging biological evolution concepts with scalable MARL approaches.

Abstract

The competitive and cooperative forces of natural selection have driven the evolution of intelligence for millions of years, culminating in nature's vast biodiversity and the complexity of human minds. Inspired by this process, we propose a novel multi-agent reinforcement learning framework where each agent is assigned a genotype and where reward functions are modelled after the concept of inclusive fitness. An agent's genetic material may be shared with other agents, and our inclusive reward function naturally accounts for this. We study the resulting social dynamics in two types of network games with prisoner's dilemmas and find that our results align with well-established principles from biology, such as Hamilton's rule. Furthermore, we outline how this framework can extend to more open-ended environments with spatial and temporal structure, finite resources, and evolving populations. We hypothesize the emergence of an arms race of strategies, where each new strategy is a gradual improvement over earlier adaptations of other agents, effectively producing a multi-agent autocurriculum analogous to biological evolution. In contrast to the binary team-based structures prevalent in earlier research, our gene-based reward structure introduces a spectrum of cooperation ranging from full adversity to full cooperativeness based on genetic similarity, enabling unique non team-based social dynamics. For example, one agent having a mutual cooperative relationship with two other agents, while the two other agents behave adversarially towards each other. We argue that incorporating inclusive fitness in agents provides a foundation for the emergence of more strategically advanced and socially intelligent agents.

Figures (5)

• Figure 1: A prisoner's dilemma played by two players with genotypes [1, 1, 1, 1] and [1, 1, 1, 0] becomes a harmony game under the inclusive reward.
• Figure 2: Example network with community structure. Here, the probability of a connection between agents inside a community is $p_{in} = 0.9$, while the probability for agents between communities is $p_{out} =0.1$. All three communities represent a separate genotype. Figure adapted from girvan2002community.
• Figure 3: Frequency of cooperation in function of the Hamming similarity with the opponent on a fully connected network (genotype length 6, 2 variants per gene) with 64 agents, one per unique genotype. The cost-benefit payoff ratio $c/b$ influences the Hamming similarity threshold at which agents start cooperating, matching Hamilton's rule hamilton1964.
• Figure 4: Proportion of cooperators in the (converged) population in function of the benefit to cost ratio $b/c$ in a random partition network (genotype length 3, 2 variants per gene). Each community represents one of $8$ unique genotypes, with 8 agents per genotype/community. $\eta$ is the network dispersal coefficient. $\langle k \rangle \! = \! 9$ for all experiments, which together with $\eta$ determines $p_{in}$ and $p_{out}$ (Eq. \ref{['eq:avk']}). Blue, red and green values show the proportion of cooperators under the inclusive reward for different dispersal coefficients. Gray values show results without inclusiveness. Cooperation increases under the inclusive reward and under small dispersal coefficients.
• Figure 5: Impression of the Neural MMO environment suarez2021neural. Rocks, water, grass and forest cover the area. Agents, represented by three connected nodes, gather food and water, and engage in combat with each other or with environmental threats.