Investigating the Impact of Direct Punishment on the Emergence of Cooperation in Multi-Agent Reinforcement Learning Systems

TL;DR

The paper investigates how direct punishment influences the emergence of cooperation in multi-agent reinforcement learning (MARL) systems and how this interacts with reputation and partner selection. It uses Iterated Prisoner's Dilemma with staged introductions of social mechanisms and trains agents via Deep Q-Networks to learn partner selection, play, and punishment under different configurations. Key findings show that third-party punishment yields higher equilibrium cooperation, direct punishment achieves higher global rewards, and combining direct and third-party punishment with partner selection and reputation yields the fastest convergence and strongest cooperation. The results inform the design of cooperative AI systems and highlight the trade-offs between cooperation and welfare, along with limitations such as assumptions about reputations and the potential for unjust punishment.

Abstract

Solving the problem of cooperation is fundamentally important for the creation and maintenance of functional societies. Problems of cooperation are omnipresent within human society, with examples ranging from navigating busy road junctions to negotiating treaties. As the use of AI becomes more pervasive throughout society, the need for socially intelligent agents capable of navigating these complex cooperative dilemmas is becoming increasingly evident. Direct punishment is a ubiquitous social mechanism that has been shown to foster the emergence of cooperation in both humans and non-humans. In the natural world, direct punishment is often strongly coupled with partner selection and reputation and used in conjunction with third-party punishment. The interactions between these mechanisms could potentially enhance the emergence of cooperation within populations. However, no previous work has evaluated the learning dynamics and outcomes emerging from Multi-Agent Reinforcement Learning (MARL) populations that combine these mechanisms. This paper addresses this gap. It presents a comprehensive analysis and evaluation of the behaviors and learning dynamics associated with direct punishment, third-party punishment, partner selection, and reputation. Finally, we discuss the implications of using these mechanisms on the design of cooperative AI systems.

Figures (19)

• Figure 1: Each episode in a simulation consists of up to three distinct stages. In the first stage, depending on whether partner selection is being studied in the current simulation, agents either select their next interaction partner using their partner selection DQN model or their partner is selected randomly out of all other agents in the population. Agents then play the Prisoner's Dilemma with their partner in the second stage before, choosing whether or not to carry out punishment in the third stage. The third stage can consist of agents performing direct punishment, third-party punishment or both direct and third-party punishment depending on the combination of social mechanisms being studied in the current simulation. The second and third stages repeat consecutively for each round in the episode, while the first stage occurs only once at the start of an episode.
• Figure 2: Each agent consists of up to three independent DQN models, each specialized for a different agent ability.
• Figure 3: Populations using direct punishment or direct punishment with partner selection learn to cooperate, but converge to a lower proportion of cooperation compared to populations using third-party punishment or both direct and third-party punishment. Despite this, populations using direct punishment achieve significantly higher levels of societal reward at convergence. This indicates that populations combining direct punishment with partner selection are most effective at maximizing global welfare through cooperation.
• Figure 4: Populations using direct punishment experience an early period of pervasive unjust punishment prior to the 140th episode, resulting in a decrease in societal reputation. As populations begin to learn to perform just punishment and cooperate, the societal reputation of populations using direct punishment increases but, to a lesser extent compared to populations using third-party punishment or combined third-party and direct punishment.
• Figure 5: Punishment per episode. Populations using direct punishment initially have the highest levels of punishment, despite also having the lowest levels of just punishment. As agents begin to learn how to punish justly, the proportion of punishment in the population increases as the act of punishing becomes rewarding for agents. As the levels of cooperation increase within a population, the use of punishment decreases regardless of the social mechanisms used as there are fewer opportunities for profitable just punishment. Populations using direct punishment have the lowest levels of punishment at convergence, resulting in reduced normative pressure to cooperate and therefore, lower levels of cooperation overall.