Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simulation-Free PSRO: Removing Game Simulation from Policy Space Response Oracles

Yingzhuo Liu, Shuodi Liu, Weijun Luo, Liuyu Xiang, Zhaofeng He

TL;DR

The paper identifies Game Simulation (GS) as the primary PSRO bottleneck in multi-agent zero-sum games and proposes Simulation-Free PSRO (SF-PSRO) to remove GS. It introduces a Dynamic Window mechanism that maintains a fixed-size strategy window and uses Nash Clustering to prune weak strategies, enabling MSS and BRS to operate without a full meta-payoff matrix. The approach achieves competitive exploitability and TrueSkill performance while dramatically reducing runtime, and demonstrates compatibility with existing SF-PSRO variants like MRCP and PSD-based BD methods. Limitations include the need to tune the window size and rely on a sketchy meta-payoff for elimination, suggesting avenues for future refinement and broader applicability.

Abstract

Policy Space Response Oracles (PSRO) combines game-theoretic equilibrium computation with learning and is effective in approximating Nash Equilibrium in zero-sum games. However, the computational cost of PSRO has become a significant limitation to its practical application. Our analysis shows that game simulation is the primary bottleneck in PSRO's runtime. To address this issue, we conclude the concept of Simulation-Free PSRO and summarize existing methods that instantiate this concept. Additionally, we propose a novel Dynamic Window-based Simulation-Free PSRO, which introduces the concept of a strategy window to replace the original strategy set maintained in PSRO. The number of strategies in the strategy window is limited, thereby simplifying opponent strategy selection and improving the robustness of the best response. Moreover, we use Nash Clustering to select the strategy to be eliminated, ensuring that the number of strategies within the strategy window is effectively limited. Our experiments across various environments demonstrate that the Dynamic Window mechanism significantly reduces exploitability compared to existing methods, while also exhibiting excellent compatibility. Our code is available at https://github.com/enochliu98/SF-PSRO.

Simulation-Free PSRO: Removing Game Simulation from Policy Space Response Oracles

TL;DR

The paper identifies Game Simulation (GS) as the primary PSRO bottleneck in multi-agent zero-sum games and proposes Simulation-Free PSRO (SF-PSRO) to remove GS. It introduces a Dynamic Window mechanism that maintains a fixed-size strategy window and uses Nash Clustering to prune weak strategies, enabling MSS and BRS to operate without a full meta-payoff matrix. The approach achieves competitive exploitability and TrueSkill performance while dramatically reducing runtime, and demonstrates compatibility with existing SF-PSRO variants like MRCP and PSD-based BD methods. Limitations include the need to tune the window size and rely on a sketchy meta-payoff for elimination, suggesting avenues for future refinement and broader applicability.

Abstract

Policy Space Response Oracles (PSRO) combines game-theoretic equilibrium computation with learning and is effective in approximating Nash Equilibrium in zero-sum games. However, the computational cost of PSRO has become a significant limitation to its practical application. Our analysis shows that game simulation is the primary bottleneck in PSRO's runtime. To address this issue, we conclude the concept of Simulation-Free PSRO and summarize existing methods that instantiate this concept. Additionally, we propose a novel Dynamic Window-based Simulation-Free PSRO, which introduces the concept of a strategy window to replace the original strategy set maintained in PSRO. The number of strategies in the strategy window is limited, thereby simplifying opponent strategy selection and improving the robustness of the best response. Moreover, we use Nash Clustering to select the strategy to be eliminated, ensuring that the number of strategies within the strategy window is effectively limited. Our experiments across various environments demonstrate that the Dynamic Window mechanism significantly reduces exploitability compared to existing methods, while also exhibiting excellent compatibility. Our code is available at https://github.com/enochliu98/SF-PSRO.
Paper Structure (22 sections, 1 theorem, 2 equations, 9 figures, 2 tables, 5 algorithms)

This paper contains 22 sections, 1 theorem, 2 equations, 9 figures, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Simulation-Free PSRO
  3. Meta-Strategy Solver in Simulation-Free PSRO
  4. Best-Response Solver in Simulation-Free PSRO
  5. Methods
  6. Challenges with Existing Methods
  7. Challenge 1: Optimizing MSS
  8. Challenge 2: Optimizing BRS
  9. Dynamic Window
  10. Experiments
  11. Main Results
  12. Ablation Study
  13. Compatibility Analysis
  14. Additional Findings
  15. Conclusions
  16. ...and 7 more sections

Key Result

Theorem 1

When the size of each player's strategy set is bounded by a constant $K$, the PSRO algorithm converges to a fixed point $(S_1^*, S_2^*, \dots, S_n^*)$ in a finite number of iterations.

Figures (9)

  • Figure 1: Game Simulation is the primary bottleneck of PSRO in terms of computational cost
  • Figure 2: Comparison between Vanilla PSRO and Simulation-Free PSRO
  • Figure 3: An illustration of the dynamic window mechanism. The left side shows each iteration, where a best-response (new strategy) is added to the strategy window, and one outdated strategy (eliminated strategy) is eliminated. The remaining ones are active strategies. The middle side depicts the three steps (❶ Filling, ❷ Nash Clustering, and ❸ Elimination) used to identify the strategy to be eliminated. The right side details the implementation of the key step, Filling.
  • Figure 4: Exploitability of Leduc Poker and Goofspiel with 2e5 and 3e5 episodes for training BR
  • Figure 5: Exploitability v.s. time cost trade-offs across Leduc Poker and Goofspiel
  • ...and 4 more figures

Theorems & Definitions (2)

  • Definition 1
  • Theorem 1: Convergence of PSRO under a Bounded Strategy Set