Policy Abstraction and Nash Refinement in Tree-Exploiting PSRO

TL;DR

This work extends Tree-Exploiting PSRO (TE-PSRO) to complex imperfect-information games by introducing abstract policies that DRL learns and uses as edges in an extensive-form empirical game, enabling scalable policy exploration. A second advance replaces NE-based best-response guidance with subgame-perfect equilibria (SPE) through a generalized backward induction (GBI) algorithm, implemented with a modular SPE solver. Empirical results on Bargain and GenGoof show that using SPE as a meta-strategy solver accelerates convergence and remains tractable in time and memory, with intermediate levels of tree augmentation (M) often performing best. The findings suggest that leveraging extensive-form refinements can improve strategy discovery in complex multi-agent environments, and that selectively expanding the empirical game tree helps manage growth without sacrificing performance.

Abstract

Policy Space Response Oracles (PSRO) interleaves empirical game-theoretic analysis with deep reinforcement learning (DRL) to solve games too complex for traditional analytic methods. Tree-exploiting PSRO (TE-PSRO) is a variant of this approach that iteratively builds a coarsened empirical game model in extensive form using data obtained from querying a simulator that represents a detailed description of the game. We make two main methodological advances to TE-PSRO that enhance its applicability to complex games of imperfect information. First, we introduce a scalable representation for the empirical game tree where edges correspond to implicit policies learned through DRL. These policies cover conditions in the underlying game abstracted in the game model, supporting sustainable growth of the tree over epochs. Second, we leverage extensive form in the empirical model by employing refined Nash equilibria to direct strategy exploration. To enable this, we give a modular and scalable algorithm based on generalized backward induction for computing a subgame perfect equilibrium (SPE) in an imperfect-information game. We experimentally evaluate our approach on a suite of games including an alternating-offer bargaining game with outside offers; our results demonstrate that TE-PSRO converges toward equilibrium faster when new strategies are generated based on SPE rather than Nash equilibrium, and with reasonable time/memory requirements for the growing empirical model.

Figures (18)

• Figure 1: Basic PSRO loop. In each iteration (or epoch), an empirical game model is extended, based on best responses (BRs) to target profile $\bm{\Tilde{\sigma}^*}$ derived from the current empirical game by the solver MSS. The BRs are computed using deep RL applied to the game simulator. EVAL is a solver (not necessarily the MSS) applied to a model to assess its quality.
• Figure 2: Part of game tree llustrating the effect of Player 1's decision $R$ on Player 2's infoset structure.
• Figure 3: Total number of information sets of both players in empirical game $\hat{G}$ over the course of TE-PSRO's runtime, averaged over all combinations of all parameters (except $M$) and seeds.
• Figure 4: Average worst-case subgame regret of NE and SPE solutions to empirical game $\hat{G}$ for the two games studied. Note that the scale of vertical axis of (b) is finer than that of (a) by a factor $\approx10^3$ while the corresponding factor $\approx10$ for (d) and (c).
• Figure 5: Average regret of solution $\bm{\sigma}^*$ of empirical game for Bargain over iterations of TE-PSRO, using NE or SPE as the MSS or EVAL and different values of $M$.

Theorems & Definitions (3)

• definition 1
• definition 2: selten75
• lemma 1