QPLEX: Duplex Dueling Multi-Agent Q-Learning

TL;DR

QPLEX introduces a duplex dueling multi-agent Q-learning framework that enforces an Advantage-Based IGM to achieve complete IGM expressiveness under CTDE. The Transformation and Dueling Mixing components enable scalable joint action-value factorization, with positive attention-based credit weights λ_i coordinating per-agent advantages to form the joint advantage. Empirically, QPLEX delivers state-of-the-art performance on StarCraft II micromanagement in both online and offline data regimes and demonstrates enhanced stability and sample efficiency compared to baselines like QMIX, VDN, QTRAN, and QATTEN. The approach shows promise for offline MARL and motivates future work on continuous-action extensions and broader offline-data utilization.

Abstract

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

Figures (18)

• Figure 1: (a) The dueling mixing network structure. (b) The overall QPLEX architecture. (c) Agent network structure (bottom) and Transformation network structure (top).
• Figure 2: (a) Payoff matrix for a harder one-step game. Boldface means the optimal joint action selection from the payoff matrix. The strikethroughs indicate the original matrix game proposed by QTRAN. (b) The learning curves of QPLEX and other baselines. (c) The learning curve of QPLEX, whose suffix $a$L$b$H denotes the neural network size with $a$ layers and $b$ heads (multi-head attention) for learning importance weights $\lambda_i$ (see Eq. (\ref{['eq:Atot']}) and (\ref{['eq:lambda']})), respectively.
• Figure 3: (a) A special two-state MMDP used to demonstrate the training stability of the multi-agent Q-learning algorithms. $r$ is a shorthand for $r(s, \bm{a})$. (b) The learning curves of $\|Q_{tot}\|_\infty$ in a specific two-state MMDP.
• Figure 4: (a) The median test win %, averaged across all 17 scenarios. (b) The number of scenarios in which the algorithms' median test win % is the highest by at least 1/32 (smoothed).
• Figure 5: Learning curves of StarCraft II with online data collection.

Theorems & Definitions (9)

• Definition 1: Advantage-based IGM
• Proposition 1
• Proposition 2
• Definition 1: Advantage-based IGM
• Proposition 2
• proof
• proof
• Proposition 2
• proof