Self-adaptive PSRO: Towards an Automatic Population-based Game Solver

TL;DR

This work tackles the difficulty of manually tuning hyperparameters in Policy-Space Response Oracles (PSRO) for two-player zero-sum games by introducing Parametric PSRO (PPSRO) and Self-adaptive PSRO (SPSRO). PPSRO unifies gradient descent ascent and various PSRO variants through game-free solver weights $\boldsymbol{\alpha}$ and game-based BR hyperparameters $(\beta, K)$, while SPSRO learns an HPO policy to adapt these values online. To overcome online HPO limitations, the authors develop a novel offline Transformer-based HPO method that models hyperparameter selection as sequence modeling, enabling transfer to new games. Experiments across normal-form and extensive-form games show that SPSRO with Transformer improves learning performance over baselines and demonstrates better generalization, suggesting a practical, plug-and-play approach to automatic hyperparameter optimization in PSRO. The approach potentially broadens PSRO applicability by reducing manual tuning and enabling robust, cross-game adaptation of meta-solvers and BR training strategies, with $y^e=\mathcal{R}(\boldsymbol{\sigma}^e)/\mathcal{R}(\boldsymbol{\sigma}^{1})+h^e/h^{1}$ guiding the HPO objective.

Abstract

Policy-Space Response Oracles (PSRO) as a general algorithmic framework has achieved state-of-the-art performance in learning equilibrium policies of two-player zero-sum games. However, the hand-crafted hyperparameter value selection in most of the existing works requires extensive domain knowledge, forming the main barrier to applying PSRO to different games. In this work, we make the first attempt to investigate the possibility of self-adaptively determining the optimal hyperparameter values in the PSRO framework. Our contributions are three-fold: (1) Using several hyperparameters, we propose a parametric PSRO that unifies the gradient descent ascent (GDA) and different PSRO variants. (2) We propose the self-adaptive PSRO (SPSRO) by casting the hyperparameter value selection of the parametric PSRO as a hyperparameter optimization (HPO) problem where our objective is to learn an HPO policy that can self-adaptively determine the optimal hyperparameter values during the running of the parametric PSRO. (3) To overcome the poor performance of online HPO methods, we propose a novel offline HPO approach to optimize the HPO policy based on the Transformer architecture. Experiments on various two-player zero-sum games demonstrate the superiority of SPSRO over different baselines.

Figures (18)

• Figure 1: NashConv of different PSRO runs.
• Figure 2: Illustration of Self-adaptive Policy-Space Response Oracles (SPSRO). PSRO and PPSRO are two special cases of SPSRO. Illustration inspired by muller2020generalized.
• Figure 3: HPO based on Transformer. At each epoch, the Transformer model predicts parameter values in an autoregressive manner using a causal self-attention mask, i.e., each predicted parameter value will be fed into the model to generate the next parameter value.
• Figure 4: Evaluation performance. The top and bottom rows correspond to NFGs and EFGs, respectively.
• Figure 5: NashConvs of Optuna and Transformer in different extensive-form games.