Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Periodicity in Hedge-myopic system and an asymmetric NE-solving paradigm for two-player zero-sum games

Xinxiang Guo, Yifen Mu, Xiaoguang Yang

TL;DR

This paper proposes an asymmetric paradigm for solving two-player zero-sum games and proves that the stage strategy sequence of both players are periodic after finite stages and the time-averaged strategy of player Y within one period is an exact NE strategy.

Abstract

In this paper, we consider the $n \times n$ two-payer zero-sum repeated game in which one player (player X) employs the popular Hedge (also called multiplicative weights update) learning algorithm while the other player (player Y) adopts the myopic best response. We investigate the dynamics of such Hedge-myopic system by defining a metric $Q(\textbf{x}_t)$, which measures the distance between the stage strategy $\textbf{x}_t$ and Nash Equilibrium (NE) strategy of player X. We analyze the trend of $Q(\textbf{x}_t)$ and prove that it is bounded and can only take finite values on the evolutionary path when the payoff matrix is rational and the game has an interior NE. Based on this, we prove that the stage strategy sequence of both players are periodic after finite stages and the time-averaged strategy of player Y within one period is an exact NE strategy. Accordingly, we propose an asymmetric paradigm for solving two-player zero-sum games. For the special game with rational payoff matrix and an interior NE, the paradigm can output the precise NE strategy; for any general games we prove that the time-averaged strategy can converge to an approximate NE. In comparison to the NE-solving method via Hedge self-play, this HBR paradigm exhibits faster computation/convergence, better stability and can attain precise NE convergence in most real cases.

Periodicity in Hedge-myopic system and an asymmetric NE-solving paradigm for two-player zero-sum games

TL;DR

This paper proposes an asymmetric paradigm for solving two-player zero-sum games and proves that the stage strategy sequence of both players are periodic after finite stages and the time-averaged strategy of player Y within one period is an exact NE strategy.

Abstract

In this paper, we consider the two-payer zero-sum repeated game in which one player (player X) employs the popular Hedge (also called multiplicative weights update) learning algorithm while the other player (player Y) adopts the myopic best response. We investigate the dynamics of such Hedge-myopic system by defining a metric , which measures the distance between the stage strategy and Nash Equilibrium (NE) strategy of player X. We analyze the trend of and prove that it is bounded and can only take finite values on the evolutionary path when the payoff matrix is rational and the game has an interior NE. Based on this, we prove that the stage strategy sequence of both players are periodic after finite stages and the time-averaged strategy of player Y within one period is an exact NE strategy. Accordingly, we propose an asymmetric paradigm for solving two-player zero-sum games. For the special game with rational payoff matrix and an interior NE, the paradigm can output the precise NE strategy; for any general games we prove that the time-averaged strategy can converge to an approximate NE. In comparison to the NE-solving method via Hedge self-play, this HBR paradigm exhibits faster computation/convergence, better stability and can attain precise NE convergence in most real cases.
Paper Structure (16 sections, 10 theorems, 49 equations, 9 figures, 1 algorithm)

This paper contains 16 sections, 10 theorems, 49 equations, 9 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary and Problem Formulation
  3. Online Learning and Hedge Algorithm
  4. Normal-form zero-sum Game
  5. Problem Formulation
  6. Main Results
  7. Rational Games with an Interior Equilibrium
  8. Non-periodicity Examples without Rational Interior NE
  9. A Novel NE-solving Algorithm and Experiments
  10. HBR: an Asymmetric Equilibrium-solving Paradigm
  11. Experimental Results
  12. Conclusion and Future Work
  13. Acknowledgement
  14. Proof of Proposition \ref{['Lem_interior']}
  15. Proof of Theorem \ref{['Lem_finite_values']}
  16. ...and 1 more sections

Key Result

Theorem 2.1

Assume that the loss function $\ell$ is convex in its first argument and takes values in $[0, 1]$. For any $t$ and $\eta > 0$, and for all $q_1, q_2, \cdots ,q_t \in \mathcal{Q}$, the regret for the Hedge algorithm satisfies In particular, for $\eta = \sqrt{ \frac{8\ln N }{t}}$, the upper bound becomes $\sqrt{\frac{t}{2}\ln N}$.

Figures (9)

  • Figure 1: Graphical illustration of $Z_p$, $Z_u$ and NE for $3\times 3$ games.
  • Figure 2: The evolving path of the strategy for player X. The red point represents the NE strategy of player X.
  • Figure 3: The action sequence of player Y in two different games.
  • Figure 4: The evolution of $\textbf{x}_t$ in Example \ref{['Exp_non_interior']}.
  • Figure 5: The evolution of $x_t$ in the Hedge-myopic system for Example \ref{['Exp_special_case']}.
  • ...and 4 more figures

Theorems & Definitions (24)

  • Theorem 2.1: Theorem 2.2 of cesa2006prediction
  • Theorem 3.1
  • Lemma 3.1: Lemma A.1 of cesa2006prediction
  • Lemma 3.2: Representation of Bounded Polyhedra
  • Proposition 3.1
  • Corollary 3.1
  • proof
  • proof : Proof of Theorem \ref{['Lem_Qsequence_bounded']}
  • Theorem 3.2
  • Remark 1
  • ...and 14 more