Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Nudges for Conditional Cooperation: A Multi-Agent Reinforcement Learning Model

Shatayu Kulkarni, Sabine Brunswicker

TL;DR

This paper introduces a multi-agent reinforcement learning model for public goods games, with 3 CC learning agents using aspirational reinforcement learning and 1 nudging agent using deep reinforcement learning to learn nudges that optimize cooperation.

Abstract

The public goods game describes a social dilemma in which a large proportion of agents act as conditional cooperators (CC): they only act cooperatively if they see others acting cooperatively because they satisfice with the social norm to be in line with what others are doing instead of optimizing cooperation. CCs are guided by aspiration-based reinforcement learning guided by past experiences of interactions with others and satisficing aspirations. In many real-world settings, reinforcing social norms do not emerge. In this paper, we propose that an optimizing reinforcement agent can facilitate cooperation through nudges, i.e. indirect mechanisms for cooperation to happen. The agent's goal is to motivate CCs into cooperation through its own actions to create social norms that signal that others are cooperating. We introduce a multi-agent reinforcement learning model for public goods games, with 3 CC learning agents using aspirational reinforcement learning and 1 nudging agent using deep reinforcement learning to learn nudges that optimize cooperation. For our nudging agent, we model two distinct reward functions, one maximizing the total game return (sum DRL) and one maximizing the number of cooperative contributions contributions higher than a proportional threshold (prop DRL). Our results show that our aspiration-based RL model for CC agents is consistent with empirically observed CC behavior. Games combining 3 CC RL agents and one nudging RL agent outperform the baseline consisting of 4 CC RL agents only. The sum DRL nudging agent increases the total sum of contributions by 8.22% and the total proportion of cooperative contributions by 12.42%, while the prop nudging DRL increases the total sum of contributions by 8.85% and the total proportion of cooperative contributions by 14.87%. Our findings advance the literature on public goods games and reinforcement learning.

Learning Nudges for Conditional Cooperation: A Multi-Agent Reinforcement Learning Model

TL;DR

This paper introduces a multi-agent reinforcement learning model for public goods games, with 3 CC learning agents using aspirational reinforcement learning and 1 nudging agent using deep reinforcement learning to learn nudges that optimize cooperation.

Abstract

The public goods game describes a social dilemma in which a large proportion of agents act as conditional cooperators (CC): they only act cooperatively if they see others acting cooperatively because they satisfice with the social norm to be in line with what others are doing instead of optimizing cooperation. CCs are guided by aspiration-based reinforcement learning guided by past experiences of interactions with others and satisficing aspirations. In many real-world settings, reinforcing social norms do not emerge. In this paper, we propose that an optimizing reinforcement agent can facilitate cooperation through nudges, i.e. indirect mechanisms for cooperation to happen. The agent's goal is to motivate CCs into cooperation through its own actions to create social norms that signal that others are cooperating. We introduce a multi-agent reinforcement learning model for public goods games, with 3 CC learning agents using aspirational reinforcement learning and 1 nudging agent using deep reinforcement learning to learn nudges that optimize cooperation. For our nudging agent, we model two distinct reward functions, one maximizing the total game return (sum DRL) and one maximizing the number of cooperative contributions contributions higher than a proportional threshold (prop DRL). Our results show that our aspiration-based RL model for CC agents is consistent with empirically observed CC behavior. Games combining 3 CC RL agents and one nudging RL agent outperform the baseline consisting of 4 CC RL agents only. The sum DRL nudging agent increases the total sum of contributions by 8.22% and the total proportion of cooperative contributions by 12.42%, while the prop nudging DRL increases the total sum of contributions by 8.85% and the total proportion of cooperative contributions by 14.87%. Our findings advance the literature on public goods games and reinforcement learning.
Paper Structure (17 sections, 7 equations, 5 figures, 3 tables)

This paper contains 17 sections, 7 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-agent Reinforcement Learning and Cooperation
  4. Aspiration-based Reinforcement Learning and Economic Game Theory
  5. The Public Goods Game and the Reinforcement Learning Agents
  6. Game Setup
  7. The Reinforcement Learning Agents
  8. The Aspirational Reinforcement Learning Agents
  9. The Nudging Deep Reinforcement Learning (DRL) Agent
  10. The Experiment Setup
  11. Results
  12. Validation
  13. Deep Reinforcement Learning Agent Training
  14. Game Results
  15. Conclusion
  16. ...and 2 more sections

Figures (5)

  • Figure 1: The relationship between three CC agents' contributions in round $t - 1$ and the fourth CC agents' contribution in round $t$ for CC agents in the prior study ezaki_reinforcement_2016 (a) and in this paper (b).
  • Figure 2: Mean episodic reward for both the sum (a) and prop (b) DRL agents throughout the training period of 4,000,000 games.
  • Figure 3: The mean contributions of the three other CC agents (a), the proportion of their contributions deemed cooperative when interacting with baseline agents and the DRL agents (b), and the average contributions of both baseline and DRL agents across rounds (c).
  • Figure 4: The distributions of CC agents when playing with the baseline (a), the sum DRL agent (b), and the prop DRL agent (c) at each round represented with heat maps.
  • Figure 5: The difference in frequency of contributions by CC agents when playing with the sum DRL agent (a) and prop DRL agent (b) compared to baseline agents. These heatmaps show the change in probability of following up contributions with higher or lower amounts, indicating the positive influence of both DRL agents on cooperative behavior.