Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A-PSRO: A Unified Strategy Learning Method with Advantage Function for Normal-form Games

Yudong Hu, Haoran Li, Congying Han, Tiande Guo, Mingqiang Li, Bonan Li

TL;DR

This work introduces the advantage function as an enhanced evaluation metric for strategies, enabling a unified learning objective for agents engaged in normal-form games and proves that the advantage function exhibits favorable properties and is connected with the Nash equilibrium.

Abstract

Solving Nash equilibrium is the key challenge in normal-form games with large strategy spaces, where open-ended learning frameworks offer an efficient approach. In this work, we propose an innovative unified open-ended learning framework A-PSRO, i.e., Advantage Policy Space Response Oracle, as a comprehensive framework for both zero-sum and general-sum games. In particular, we introduce the advantage function as an enhanced evaluation metric for strategies, enabling a unified learning objective for agents engaged in normal-form games. We prove that the advantage function exhibits favorable properties and is connected with the Nash equilibrium, which can be used as an objective to guide agents to learn strategies efficiently. Our experiments reveal that A-PSRO achieves a considerable decrease in exploitability in zero-sum games and an escalation in rewards in general-sum games, significantly outperforming previous PSRO algorithms.

A-PSRO: A Unified Strategy Learning Method with Advantage Function for Normal-form Games

TL;DR

This work introduces the advantage function as an enhanced evaluation metric for strategies, enabling a unified learning objective for agents engaged in normal-form games and proves that the advantage function exhibits favorable properties and is connected with the Nash equilibrium.

Abstract

Solving Nash equilibrium is the key challenge in normal-form games with large strategy spaces, where open-ended learning frameworks offer an efficient approach. In this work, we propose an innovative unified open-ended learning framework A-PSRO, i.e., Advantage Policy Space Response Oracle, as a comprehensive framework for both zero-sum and general-sum games. In particular, we introduce the advantage function as an enhanced evaluation metric for strategies, enabling a unified learning objective for agents engaged in normal-form games. We prove that the advantage function exhibits favorable properties and is connected with the Nash equilibrium, which can be used as an objective to guide agents to learn strategies efficiently. Our experiments reveal that A-PSRO achieves a considerable decrease in exploitability in zero-sum games and an escalation in rewards in general-sum games, significantly outperforming previous PSRO algorithms.
Paper Structure (32 sections, 14 theorems, 32 equations, 9 figures, 7 tables, 4 algorithms)

This paper contains 32 sections, 14 theorems, 32 equations, 9 figures, 7 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Notation and Background
  3. Symmetric Zero-Sum Games with Transitive Dimension and Cyclic Dimension
  4. General-Sum Games and Equilibrium Selection Problem
  5. Open-Ended Learning Framework
  6. Advantage Policy Space Response Oracle
  7. From Exploitability to Advantage Function
  8. A-PSRO for Solving Zero-Sum Games
  9. A-PSRO for Solving Two-Player General-Sum Games
  10. A-PSRO for Solving Multi-Player Games
  11. Experiment Results and Discussion
  12. Experiments in Symmetric Zero-Sum Games
  13. Experiments in Two-player General-Sum Games
  14. Experiments in Multi-player Games
  15. Conclusion
  16. ...and 17 more sections

Key Result

Theorem 3.1

In symmetric zero-sum games, if the joint strategy $(\pi^1,\pi^2)$ is a Nash equilibrium, we have $(\pi^1,\pi^1)$ and $(\pi^2,\pi^2)$ are both Nash equilibriums.

Figures (9)

  • Figure 1: The geometrical structure examples of zero-sum games and general-sum games. Figure (a) shows the structure of a zero-sum game with both transitive and cyclic dimensions. The direction of the strategy gradient refers to the expected updates for a strategy that maximize the reward. Figure (b) shows the structure of a general-sum game with multiple equilibia. The independent learning process of the agents leads to the update of the strategy in the direction indicated by the arrow.
  • Figure 2: The exploitability of the joint strategy learned by agents in various zero-sum games is depicted. The reduction in exploitability through population iterations can serve as an indicator of the effectiveness in approximating the Nash equilibrium.
  • Figure 3: The advantage of strategies in different games. Strategies with lighter colors have a higher advantage. In symmetric zero-sum games, the Nash equilibrium strategy has the highest advantage 0.
  • Figure 4: The running time of different algorithms.
  • Figure 5: The joint reward of the agent system in general-sum games. The Staghunt game and the RSP game are repeated 10 times and averaged for plotting. Randomly generated games contain 100 games with the same reward distribution.
  • ...and 4 more figures

Theorems & Definitions (24)

  • Theorem 3.1
  • Theorem 3.2
  • Definition 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Theorem 3.6
  • Definition 3.7
  • Definition 3.8
  • Theorem 3.9
  • Theorem 3.10
  • ...and 14 more