Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emergent Cooperation in Quantum Multi-Agent Reinforcement Learning Using Communication

Michael Kölle, Christian Reff, Leo Sünkel, Julian Hager, Gerhard Stenzel, Claudia Linnhoff-Popien

TL;DR

This work investigates emergent cooperation in Quantum Multi-Agent Reinforcement Learning by adapting eight classical communication mechanisms to quantum Q-learning agents and evaluating them on three sequential social dilemmas. It demonstrates that token-based, TD-guided, and consensus-equipped protocols (notably MATETD and MEDIATE variants) reliably foster cooperation, while other approaches show mixed performance. The findings imply that communication remains a viable mechanism for coordinating quantum agents in SSDs, with practical implications for scalable quantum multi-agent systems. However, experiments in a larger, more complex Harvest-like environment reveal current quantum architectures face optimization and representation challenges that limit performance relative to classical baselines.

Abstract

Emergent cooperation in classical Multi-Agent Reinforcement Learning has gained significant attention, particularly in the context of Sequential Social Dilemmas (SSDs). While classical reinforcement learning approaches have demonstrated capability for emergent cooperation, research on extending these methods to Quantum Multi-Agent Reinforcement Learning remains limited, particularly through communication. In this paper, we apply communication approaches to quantum Q-Learning agents: the Mutual Acknowledgment Token Exchange (MATE) protocol, its extension Mutually Endorsed Distributed Incentive Acknowledgment Token Exchange (MEDIATE), the peer rewarding mechanism Gifting, and Reinforced Inter-Agent Learning (RIAL). We evaluate these approaches in three SSDs: the Iterated Prisoner's Dilemma, Iterated Stag Hunt, and Iterated Game of Chicken. Our experimental results show that approaches using MATE with temporal-difference measure (MATE\textsubscript{TD}), AutoMATE, MEDIATE-I, and MEDIATE-S achieved high cooperation levels across all dilemmas, demonstrating that communication is a viable mechanism for fostering emergent cooperation in Quantum Multi-Agent Reinforcement Learning.

Emergent Cooperation in Quantum Multi-Agent Reinforcement Learning Using Communication

TL;DR

This work investigates emergent cooperation in Quantum Multi-Agent Reinforcement Learning by adapting eight classical communication mechanisms to quantum Q-learning agents and evaluating them on three sequential social dilemmas. It demonstrates that token-based, TD-guided, and consensus-equipped protocols (notably MATETD and MEDIATE variants) reliably foster cooperation, while other approaches show mixed performance. The findings imply that communication remains a viable mechanism for coordinating quantum agents in SSDs, with practical implications for scalable quantum multi-agent systems. However, experiments in a larger, more complex Harvest-like environment reveal current quantum architectures face optimization and representation challenges that limit performance relative to classical baselines.

Abstract

Emergent cooperation in classical Multi-Agent Reinforcement Learning has gained significant attention, particularly in the context of Sequential Social Dilemmas (SSDs). While classical reinforcement learning approaches have demonstrated capability for emergent cooperation, research on extending these methods to Quantum Multi-Agent Reinforcement Learning remains limited, particularly through communication. In this paper, we apply communication approaches to quantum Q-Learning agents: the Mutual Acknowledgment Token Exchange (MATE) protocol, its extension Mutually Endorsed Distributed Incentive Acknowledgment Token Exchange (MEDIATE), the peer rewarding mechanism Gifting, and Reinforced Inter-Agent Learning (RIAL). We evaluate these approaches in three SSDs: the Iterated Prisoner's Dilemma, Iterated Stag Hunt, and Iterated Game of Chicken. Our experimental results show that approaches using MATE with temporal-difference measure (MATE\textsubscript{TD}), AutoMATE, MEDIATE-I, and MEDIATE-S achieved high cooperation levels across all dilemmas, demonstrating that communication is a viable mechanism for fostering emergent cooperation in Quantum Multi-Agent Reinforcement Learning.
Paper Structure (21 sections, 5 figures)

This paper contains 21 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Communication Mechanisms
  3. Mutual Acknowledgment Token Exchange
  4. MEDIATE Extensions
  5. Gifting
  6. Reinforced Inter-Agent Learning
  7. Variational Quantum Circuit Architecture
  8. Matrix Game Architectures
  9. Harvest Game Architecture
  10. Experimental Setup
  11. Matrix Games
  12. Harvest Game
  13. Evaluation Metrics
  14. Training and Hyperparameters
  15. Results
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Comparison of different VQC architectures for our multi-agent experiments.
  • Figure 2: Comparison of performance metrics in the Iterated Prisoner’s Dilemma (IPD) across all approaches: (a) collective cumulative reward, (b) cooperation frequency, and (c) inequality between agents.
  • Figure 3: Performance metrics in the Iterated Stag Hunt (low-risk variant): (a) collective cumulative reward, (b) mutual hunt frequency, and (c) inequality between agents.
  • Figure 4: Performance metrics in the Iterated Game of Chicken: (a) collective cumulative reward, (b) frequency of mutual chicken actions, and (c) inequality between agents.
  • Figure 5: Collective cumulative reward in the Harvest Game.