Emergent Cooperation under Uncertain Incentive Alignment

TL;DR

This work investigates how cooperation emerges among reinforcement learning agents when incentive alignment is uncertain, using the Extended Public Goods Game (EPGG) to span competitive, mixed-motive, and cooperative regimes. It analyzes reputation-based social norms, steering agents, and intrinsic rewards as mechanisms to promote cooperation under uncertainty, modeling observation noise on the incentive multiplier $f$ and evaluating both tabular Q-learning and Deep Q-Networks. The key findings show that uncertainty substantially reduces cooperation in cooperative and mixed-motive settings, but reputation with an effective social norm and intrinsic rewards can restore near-optimal cooperation, especially when steering agents are present. The results highlight the importance of combining social and intrinsic motivation signals to robustly foster cooperative behavior in multi-agent systems facing uncertain incentives, with implications for scalable, cooperative AI in real-world, sparse-interaction environments.

Abstract

Understanding the emergence of cooperation in systems of computational agents is crucial for the development of effective cooperative AI. Interaction among individuals in real-world settings are often sparse and occur within a broad spectrum of incentives, which often are only partially known. In this work, we explore how cooperation can arise among reinforcement learning agents in scenarios characterised by infrequent encounters, and where agents face uncertainty about the alignment of their incentives with those of others. To do so, we train the agents under a wide spectrum of environments ranging from fully competitive, to fully cooperative, to mixed-motives. Under this type of uncertainty we study the effects of mechanisms, such as reputation and intrinsic rewards, that have been proposed in the literature to foster cooperation in mixed-motives environments. Our findings show that uncertainty substantially lowers the agents' ability to engage in cooperative behaviour, when that would be the best course of action. In this scenario, the use of effective reputation mechanisms and intrinsic rewards boosts the agents' capability to act nearly-optimally in cooperative environments, while greatly enhancing cooperation in mixed-motive environments as well.

Figures (5)

• Figure 1: Normal form games instantiating the EPGG for two players (X and Y) with 4 coins each, for four possible values of the multiplication factor: $f=0.5$ (competitive game), $f=1.0$ (boundary game with both $CC$ and $DD$ optimal), $f=1.5$ (mixed-motive game) and $f=3.5$ (cooperative game).
• Figure 2: Average cooperation for DQN agents trained across environments with different multiplication factors. The top row (a - d) shows the results in the absence of cooperation-aiding mechanisms, the middle row (e - h) in the presence of reputation mechanisms and a social norm that aids cooperation, and the bottom row (i - l) in the presence of intrinsic rewards.
• Figure 3: Average cooperation for DQN agents trained across environments with different multiplication factors under uncertainty ($\sigma_i = 2 \; \forall \; i \in N$). Results are displayed in four rows: the first without reputation or intrinsic rewards, the second with reputation and a social norm that aids cooperation, the third with the intrinsic rewards formulation, and the fourth with reputation, a social norm that aids cooperation, and the intrinsic rewards formulation.
• Figure 4: Average cooperation for Q-learning agents in four environments with different multiplication factors. Results are presented in four rows: in absence of cooperation-aiding mechanisms (a-d), with the inclusion of the reputation mechanism (e - h), with the intrinsic rewards mechanism (i - l), and with reputation and intrinsic rewards formulation (m - p).
• Figure 5: Average cooperation for DQN agents trained across environments with different multiplication factors, with the addition of reputation, a social norm that aids cooperation, and the intrinsic rewards formulation.