Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, Shih-wei Liao

TL;DR

The paper investigates how implementation details and the monotonicity constraint influence performance in cooperative multi-agent RL, focusing on QMIX and its relatives. It shows that code-level optimizations can drive large gains and that, when standardized, QMIX often outperforms monotonicity-relaxing variants in SMAC and DEPP. Additionally, it provides a theoretical and empirical view that monotonicity can improve sample efficiency in purely cooperative settings, supported by RIIT and VMIX analyses. The work culminates with practical guidance and an open-source release to foster fair benchmarking and further study.

Abstract

Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.

Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

TL;DR

The paper investigates how implementation details and the monotonicity constraint influence performance in cooperative multi-agent RL, focusing on QMIX and its relatives. It shows that code-level optimizations can drive large gains and that, when standardized, QMIX often outperforms monotonicity-relaxing variants in SMAC and DEPP. Additionally, it provides a theoretical and empirical view that monotonicity can improve sample efficiency in purely cooperative settings, supported by RIIT and VMIX analyses. The work culminates with practical guidance and an open-source release to foster fair benchmarking and further study.

Abstract

Many complex multi-agent systems such as robot swarms control and autonomous vehicle coordination can be modeled as Multi-Agent Reinforcement Learning (MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge (SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX target relaxing the monotonicity constraint of QMIX, allowing for performance improvement in SMAC. In this paper, we investigate the code-level optimizations of these variants and the monotonicity constraint. (1) We find that such improvements of the variants are significantly affected by various code-level optimizations. (2) The experiment results show that QMIX with normalized optimizations outperforms other works in SMAC; (3) beyond the common wisdom from these works, the monotonicity constraint can improve sample efficiency in SMAC and DEPP. We also discuss why monotonicity constraints work well in purely cooperative tasks with a theoretical analysis. We open-source the code at \url{https://github.com/hijkzzz/pymarl2}.

Paper Structure

This paper contains 35 sections, 1 theorem, 19 equations, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Related Works
  4. Experiments Setup
  5. Benchmark Environment
  6. Evaluation Metric
  7. Experiments
  8. Rethinking the Code-level Optimizations
  9. Optimizer
  10. Eligibility Traces
  11. Replay Buffer Size
  12. Rollout Process Number
  13. Hidden Size
  14. Exploration Steps
  15. Overall Impacts
  16. ...and 20 more sections

Key Result

Proposition 1

Purely Cooperative Tasks can be represented by monotonic mixing networks.

Figures (11)

  • Figure 1: Adam significantly improves performance when samples are updated quickly.
  • Figure 2: Experiments for Q($\lambda$).
  • Figure 4: Setting the replay buffer size to 5000 episodes allows for QMIX's learning to be more stable than by setting it to 20000 episodes.
  • Figure 5: Given the total number of samples, fewer processes achieve better performance. We set the replay buffer size to be proportional to the number of processes to ensure that the novelty of the samples is consistent.
  • Figure 6: On the hard scenario $3s5z\_vs\_3s6z$, increasing the width of neural network significantly improves the performance of QMIX.
  • ...and 6 more figures

Theorems & Definitions (6)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Proposition 1
  • proof