FM3Q: Factorized Multi-Agent MiniMax Q-Learning for Two-Team Zero-Sum Markov Game

TL;DR

This work tackles learning in two-team zero-sum Markov games (2t0sMGs) by introducing the IGMM principle, which aligns global minimax behavior with individual greedy actions. Building on this, FM3Q factorizes the joint minimax Q function into per-agent components $Q_i^+$ and $Q_j^-$ and employs a monotonic mixing network to ensure consistent decision-making under decentralization, with a fitted-Q-iteration–style online learning procedure and theoretical convergence guarantees. The authors demonstrate convergence to a superb Q function $Q_{tot}^*$ and show through experiments on Pong-like Wimblepong, MPE, and RoboMaster that FM3Q achieves superior data efficiency, faster learning, and lower exploitability compared with strong baselines. The approach advances practical, scalable coordination and competition in 2t0sMGs, offering a principled path toward decentralized minimax policies in complex multi-agent settings.

Abstract

Many real-world applications involve some agents that fall into two teams, with payoffs that are equal within the same team but of opposite sign across the opponent team. The so-called two-team zero-sum Markov games (2t0sMGs) can be resolved with reinforcement learning in recent years. However, existing methods are thus inefficient in light of insufficient consideration of intra-team credit assignment, data utilization and computational intractability. In this paper, we propose the individual-global-minimax (IGMM) principle to ensure the coherence between two-team minimax behaviors and the individual greedy behaviors through Q functions in 2t0sMGs. Based on it, we present a novel multi-agent reinforcement learning framework, Factorized Multi-Agent MiniMax Q-Learning (FM3Q), which can factorize the joint minimax Q function into individual ones and iteratively solve for the IGMM-satisfied minimax Q functions for 2t0sMGs. Moreover, an online learning algorithm with neural networks is proposed to implement FM3Q and obtain the deterministic and decentralized minimax policies for two-team players. A theoretical analysis is provided to prove the convergence of FM3Q. Empirically, we use three environments to evaluate the learning efficiency and final performance of FM3Q and show its superiority on 2t0sMGs.

Figures (7)

• Figure 1: The overall architecture of FM3Q, where two-team agents acquire observations from the environment and derive their actions and corresponding individual Q-values through their utility modules. The individual Q-values of the Ant undergo the negation process and, combined with the individual Q-values of the Pro, are processed through a mixing network to yield their minimax joint Q-values. The black arrows indicate that FM3Q interacts with the environment and stores experience when performing execution. The purple arrows indicate the foward propagation process when training model. The red arrow show the gradient flow.
• Figure 2: Illustrations of Wimblepong, MPE, and RoboMaster. (a) A green and a red paddle represent two players, respectively, and a white circle represents a ball. (b) Three blue agents and three red agents move, collide, and obstruct each other to maneuver their teams as close as possible to a green target. (c) There are two blue robots, two red robots, and nine obstacles.
• Figure 3: Illustration of the performance of FM3Q during training by evaluating against the rule-based bots.
• Figure 4: Illustration of the Round-Robin results of FM3Q and baselines. (Top) normalised RR returns during training. (Bottom) normalised RR returns at the end of training.
• Figure 5: Illustration of the approximate NashConv of FM3Q and baselines during training.

Theorems & Definitions (6)

• Theorem 1
• Definition 1
• Lemma 1
• Theorem 2
• Remark 1
• Remark 2