Learning Fair Cooperation in Mixed-Motive Games with Indirect Reciprocity

TL;DR

This work investigates how indirect reciprocity and reputational norms can sustain cooperation and fairness in a population split into two groups, using a donation game with $b>c>0$. It jointly analyzes an analytical evolutionary game-theory framework to identify stable norm–strategy configurations (NSS) and a learning-based approach with independent Q-learning agents to assess convergence to those equilibria. Key findings show that a defecting majority can induce minority defection, while in-group and out-group norms can steer systems toward fair or unfair cooperation; however, convergence in RL is sensitive to norm choice, $b/c$ ratio, and initial Q-values, with norms like SternJudging providing robust outcomes under certain conditions. The results highlight that, in heterogeneous populations with reputations, selecting interaction norms is crucial to address both cooperation and fairness, offering guidance for designing fair multi-agent systems and informing norm-emergence research.

Abstract

Altruistic cooperation is costly yet socially desirable. As a result, agents struggle to learn cooperative policies through independent reinforcement learning (RL). Indirect reciprocity, where agents consider their interaction partner's reputation, has been shown to stabilise cooperation in homogeneous, idealised populations. However, more realistic settings are comprised of heterogeneous agents with different characteristics and group-based social identities. We study cooperation when agents are stratified into two such groups, and allow reputation updates and actions to depend on group information. We consider two modelling approaches: evolutionary game theory, where we comprehensively search for social norms (i.e., rules to assign reputations) leading to cooperation and fairness; and RL, where we consider how the stochastic dynamics of policy learning affects the analytically identified equilibria. We observe that a defecting majority leads the minority group to defect, but not the inverse. Moreover, changing the norms that judge in and out-group interactions can steer a system towards either fair or unfair cooperation. This is made clearer when moving beyond equilibrium analysis to independent RL agents, where convergence to fair cooperation occurs with a narrower set of norms. Our results highlight that, in heterogeneous populations with reputations, carefully defining interaction norms is fundamental to tackle both dilemmas of cooperation and of fairness.

Figures (6)

• Figure 1: An illustration of an interaction between two agents which is observed by a third party using a social norm to update reputations. The donor's strategy determines the action taken based on the agents' relation to each other, and the reputation of the donor. The third party observes the action and context to assign the donor a new reputation based on a fixed social norm.
• Figure 2: We evaluate all $2^{16}$ NSS (Norm-Strategy-Strategy) combinations of norms and strategies used by the two groups. We categorise every stable NSS according to the cooperative and discriminatory nature of strategies involved: strategies can defect regardless of the opponent type (Always defect), ignore group identity (Group-agnostic) or discriminate based on group identity (Discriminatory). We observe that group-agnostic strategies are unable to coexist with any other type of strategy, and a defecting majority playing AllD leads a minority to defect -- but not the inverse. Parameters used: $b/c=5$, error rate $= 0.01$. In the appendix (Figure 1) we confirm that the number of stable states remains unchanged for a wide range of error rates and $b/c$.
• Figure 3: Fair norms (plotted as squares), which assign reputations independently of group identities, lead to fairness, but not necessarily high levels of cooperation (top-left). High cooperation and fairness can be stable with both fair and unfair norms (top-right quadrant). Parameters used: same as Figure \ref{['fig:stable-table']}.
• Figure 4: We use abbreviations from Table \ref{['tab:norms']} to refer to the norms. Diagonal entries are previously studied "fair" norms. Overall, the majority group is most affected by the in-group norm, and vice-versa for the minority group. In-group-ImageScoring causes a total cooperation breakdown, whereas in-group-Shunning is most impacted when paired with out-group-Shunning, as agents have fewer ways to recover reputation.
• Figure 5: Some norms can sustain fair and cooperative equilibria, yet may be ineffective at guiding a population of independent RL agents to converge towards such states even with an elevated $b/c$ ratio of 10, a trend which this figure demonstrates. We use different colours to represent different highly cooperative and fair norms (from the top-right quadrant of Figure \ref{['fig:coop-fair-scatter']}). Circles represent EGT results, while squares represent RL results. For some norms, RL agents are unable to converge to the strategies that theoretically form an equilibrium, which leads to lower levels of fairness, cooperation, or both. Some norms, such as SternJudging (and variations) are impacted very little, and thus indicate that strictness is required for independent RL agents to sustain fairness and cooperation. In Section C of the appendix, we explore which strategies are learned to lead to these outcomes.