Prediction Accuracy of Learning in Games : Follow-the-Regularized-Leader meets Heisenberg

TL;DR

This study analyzes the Follow-the-Regularized-Leader (FTRL) algorithm in two-player zero-sum games, providing growth rates of covariance information for continuous-time FTRL, as well as its two canonical discretization methods (Euler and Symplectic).

Abstract

We investigate the accuracy of prediction in deterministic learning dynamics of zero-sum games with random initializations, specifically focusing on observer uncertainty and its relationship to the evolution of covariances. Zero-sum games are a prominent field of interest in machine learning due to their various applications. Concurrently, the accuracy of prediction in dynamical systems from mechanics has long been a classic subject of investigation since the discovery of the Heisenberg Uncertainty Principle. This principle employs covariance and standard deviation of particle states to measure prediction accuracy. In this study, we bring these two approaches together to analyze the Follow-the-Regularized-Leader (FTRL) algorithm in two-player zero-sum games. We provide growth rates of covariance information for continuous-time FTRL, as well as its two canonical discretization methods (Euler and Symplectic). A Heisenberg-type inequality is established for FTRL. Our analysis and experiments also show that employing Symplectic discretization enhances the accuracy of prediction in learning dynamics.

Figures (15)

• Figure 1: Evolution of 400 initial conditions (blue points) for one player using (\ref{['2pMWUA']}) in R-P-S game, the red point is the expectation of blue points, the variance is calculated on the first pure strategy. At time $t=0$, these point are sampled in a small square. As time evolves, it appears that these points are randomly located on the simplex and the variance is magnified by a factor of 400. Moreover, the expectation (red point in the figures) cannot accurately predict future outcomes, as sample points in subsequent times may deviate significantly from the expected value. A demo animation can be found https://www.dropbox.com/scl/fi/u65qqyidnhzpu61mmyb7r/AltMWU1_newgif.gif?rlkey=jcuib751hn3ph69kf67hhobe4&dl=0.
• Figure 2: The two curves represent the evolution of $\Delta X_{i,\alpha}$ and $\Delta y_{i,\alpha}$ on 100 random samples of two players when they use (\ref{['2pMWUA']}). When one curve is decreasing, another curve is increasing. This implies a tradeoff between accuracy in strategy spaces versus payoff spaces.
• Figure 3: Covariance evolution of $\Delta(X_{1,1}^t)$ and $\Delta(y_{1,1}^t)$ when two players use (\ref{['2pMWUA']}) in a randomly generated game. Covariance is calculate based on sample variance of 500 randomly generated initial conditions.
• Figure 4: Covariance evolution on primal space. Calculate based on 100 random samples of the initial mixed strategies used by two players when they employ (\ref{['2pMWUA']}) in a randomly generated $3 \times 3$ game.
• Figure 5: Variance evolution of continuous FTRL, singular cases.

Theorems & Definitions (84)

• Proposition 3.1
• Proposition 4.1
• Proposition 4.2
• Theorem 5.1
• Corollary 5.2
• Corollary 5.3
• Theorem 5.4
• Lemma 1.1: Euler discretization of FTRL
• proof
• Lemma 1.2: Symplectic discretization of FTRL