A Generalized Training Approach for Multiagent Learning

TL;DR

The paper tackles the scalability of population-based multiagent training beyond two-player zero-sum games by replacing Nash with the unique, scalable solution concept $α$-Rank as the meta-solver in PSRO. It introduces the Preference-based Best Response (PBR) oracle to align PSRO updates with $α$-Rank dynamics and proves convergence guarantees in several game classes, including general-sum, many-player settings. Empirically, α-PSRO matches or outperforms exact Nash-PSRO in 2-player Kuhn and Leduc poker and scales to 3- to 5-player poker, often converging faster than approximate Nash solvers; preliminary MuJoCo soccer results demonstrate feasibility with RL oracles. Overall, the work provides a principled, scalable approach to general games that avoids NE’s computational and selection limitations, enabling broader application of PSRO in complex multiagent domains.

Abstract

This paper investigates a population-based training regime based on game-theoretic principles called Policy-Spaced Response Oracles (PSRO). PSRO is general in the sense that it (1) encompasses well-known algorithms such as fictitious play and double oracle as special cases, and (2) in principle applies to general-sum, many-player games. Despite this, prior studies of PSRO have been focused on two-player zero-sum games, a regime wherein Nash equilibria are tractably computable. In moving from two-player zero-sum games to more general settings, computation of Nash equilibria quickly becomes infeasible. Here, we extend the theoretical underpinnings of PSRO by considering an alternative solution concept, $α$-Rank, which is unique (thus faces no equilibrium selection issues, unlike Nash) and applies readily to general-sum, many-player settings. We establish convergence guarantees in several games classes, and identify links between Nash equilibria and $α$-Rank. We demonstrate the competitive performance of $α$-Rank-based PSRO against an exact Nash solver-based PSRO in 2-player Kuhn and Leduc Poker. We then go beyond the reach of prior PSRO applications by considering 3- to 5-player poker games, yielding instances where $α$-Rank achieves faster convergence than approximate Nash solvers, thus establishing it as a favorable general games solver. We also carry out an initial empirical validation in MuJoCo soccer, illustrating the feasibility of the proposed approach in another complex domain.

Figures (14)

• Figure 1: Overview of PSRO($\mathcal{M}$, $\mathcal{O}$) algorithm phases.
• Figure 2: Oracle comparisons for randomly-generated games with varying player strategy space sizes $|S^k|$. Top and bottom rows, respectively, correspond to 4- and 5-player games.
• Figure 3: Results for 2-player poker domains.
• Figure 4: Results for poker domains with more than 2 players.
• Figure A.5: \ref{['ex:psro-alpha+br']} with oracle $\mathcal{O}=\text{BR}$. In each step above, the $\alpha$-Rank support is highlighted by the light green box of the payoff table, and the BR strategy against it in bold, dark green.

Theorems & Definitions (35)

• Example 1
• Example 2
• Example 3
• Definition 1
• Definition 2
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Definition 3