Randomised Optimism via Competitive Co-Evolution for Matrix Games with Bandit Feedback

TL;DR

This work addresses learning in unknown two-player zero-sum matrix games with bandit feedback by introducing Coebl, a competitive co-evolutionary bandit learning algorithm that implements randomised optimism through evolutionary variation. The authors prove a sublinear Nash regret bound of $\tilde{O}(\sqrt{m^2T})$, extending regret analysis to evolutionary bandit learning in matrix games, and demonstrate empirical superiority over classical baselines such as Exp3, Exp3-IX, and UCB on benchmarks including Rock-Paper-Scissors, Diagonal, and BiggerNumber. The results highlight the effectiveness and robustness of randomised optimism in adversarial game settings and suggest that evolution-inspired exploration can match deterministic optimism in theory while offering practical gains. Limitations include focus on two-player zero-sum settings and specific noise assumptions, with future work proposed for general-sum/multi-player games, additional evolutionary operators, and runtime analyses.

Abstract

Learning in games is a fundamental problem in machine learning and artificial intelligence, with numerous applications~\citep{silver2016mastering,schrittwieser2020mastering}. This work investigates two-player zero-sum matrix games with an unknown payoff matrix and bandit feedback, where each player observes their actions and the corresponding noisy payoff. Prior studies have proposed algorithms for this setting~\citep{o2021matrix,maiti2023query,cai2024uncoupled}, with \citet{o2021matrix} demonstrating the effectiveness of deterministic optimism (e.g., \ucb) in achieving sublinear regret. However, the potential of randomised optimism in matrix games remains theoretically unexplored. We propose Competitive Co-evolutionary Bandit Learning (\coebl), a novel algorithm that integrates evolutionary algorithms (EAs) into the bandit framework to implement randomised optimism through EA variation operators. We prove that \coebl achieves sublinear regret, matching the performance of deterministic optimism-based methods. To the best of our knowledge, this is the first theoretical regret analysis of an evolutionary bandit learning algorithm in matrix games. Empirical evaluations on diverse matrix game benchmarks demonstrate that \coebl not only achieves sublinear regret but also consistently outperforms classical bandit algorithms, including \exptr~\citep{auer2002nonstochastic}, the variant \exptrni~\citep{cai2024uncoupled}, and \ucb~\citep{o2021matrix}. These results highlight the potential of evolutionary bandit learning, particularly the efficacy of randomised optimism via evolutionary algorithms in game-theoretic settings.

Figures (10)

• Figure 1: Regret and KL-divergence for Self-Plays on RPS games
• Figure 2: Regret for $\textsc{Alg}\xspace~1$-vs-$\textsc{Alg}\xspace~2$ on RPS games
• Figure 3: Regret and TV Distance for Self-Plays on Diagonal
• Figure 4: Regret for $\textsc{Alg}\xspace~1$-vs-$\textsc{Alg}\xspace~2$ on Diagonal.
• Figure 5: Regret and TV Distance for Self-Plays on BiggerNumber

Theorems & Definitions (14)

• Definition 1
• Definition 2
• Definition 3: Nash Regret o2021matrix
• Lemma 0
• Theorem 1: Main Result
• proof : Sketch of Proof
• Lemma 1
• proof : Proof of Lemma \ref{['lem:identityOne']}
• Lemma 1
• proof : Proof of Lemma \ref{['lem:identityTwo']}