On the Stability of Learning in Network Games with Many Players

TL;DR

This work studies the Q-Learning Dynamics, a classical model for exploration and exploitation in multi-agent learning, and shows that independent learning dynamics may converge to approximate Nash Equilibrium, even in the presence of many agents.

Abstract

Multi-agent learning algorithms have been shown to display complex, unstable behaviours in a wide array of games. In fact, previous works indicate that convergent behaviours are less likely to occur as the total number of agents increases. This seemingly prohibits convergence to stable strategies, such as Nash Equilibria, in games with many players. To make progress towards addressing this challenge we study the Q-Learning Dynamics, a classical model for exploration and exploitation in multi-agent learning. In particular, we study the behaviour of Q-Learning on games where interactions between agents are constrained by a network. We determine a number of sufficient conditions, depending on the game and network structure, which guarantee that agent strategies converge to a unique stable strategy, called the Quantal Response Equilibrium (QRE). Crucially, these sufficient conditions are independent of the total number of agents, allowing for provable convergence in arbitrarily large games. Next, we compare the learned QRE to the underlying NE of the game, by showing that any QRE is an $ε$-approximate Nash Equilibrium. We first provide tight bounds on $ε$ and show how these bounds lead naturally to a centralised scheme for choosing exploration rates, which enables independent learners to learn stable approximate Nash Equilibrium strategies. We validate the method through experiments and demonstrate its effectiveness even in the presence of numerous agents and actions. Through these results, we show that independent learning dynamics may converge to approximate Nash Equilibria, even in the presence of many agents.

Figures (12)

• Figure 1: Examples of networks with five agents and associated $\left\lVert G \right\rVert_\infty$ and $\left\lVert G \right\rVert_2$.
• Figure 2: Lower Bound on sufficient exploration as defined by (Top) (\ref{['eqn::infty-cond']}) in a Full Network and Ring Network (Bottom) (\ref{['eqn::2-cond']}) in a Full Network, Star Network and Full Network.
• Figure 3: Lower Bound on sufficient exploration as defined by (\ref{['eqn::influence-cond']}), (\ref{['eqn::infty-cond']}) and (\ref{['eqn::2-cond']}) in a Star Network. For (\ref{['eqn::influence-cond']}), $\max_{k \in \mathcal{N}} \delta_k |\mathcal{N}_k|$ is depicted which therefore coincides with the condition defined in hussain:aamas.
• Figure 4: Trajectories of Q-Learning in a three agent Network Chakraborty Game with $\alpha = 7, \beta = 8.5$. Axes denote the probabilities with which each player chooses their first action.
• Figure 5: Q-Learning in the (Top) Network Shapley Game (Bottom) Network Sato Game with 15 agents. The boxplot depicts the probabilities with which three of the agents play their first action in the final 2500 iterations of learning. This is depicted for varying choices of exploration rate $T$.

Theorems & Definitions (31)

• Definition 2.1: Nash Equilibrium (NE)
• Definition 2.2: Quantal Response Equilibrium (QRE)
• Proposition 2.3: melo:qre
• Definition 2.4: Influence Bound
• Definition 2.5: Intensity of Identical Interests
• Theorem 3.1
• Remark 3.2
• Remark 3.3
• Remark 3.4
• Corollary 3.5