Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Playing Markov Games Without Observing Payoffs

Daniel Ablin, Alon Cohen

TL;DR

The paper tackles learning in zero-sum symmetric Markov games where payoff observations are unavailable and the opponent may have full knowledge of the game. It introduces three symmetry notions—per-state symmetric games (SSG), symmetry w.r.t. Markov policies (MSG), and symmetry w.r.t. history-dependent policies (HSG)—and shows deep structural reductions: MSG is equivalent to SSG via the EPR transformation, while HSG reduces to a single matrix game; these reductions yield tractable, polynomial-time strategies. The main results provide sublinear regret guarantees: in SSG (and thus MSG) theCopycat strategy achieves a bound of $O\left(n\sqrt{T|S|H}\right)$, and in HSG a bound of $O(Hn\sqrt{T})$, with a matching lower bound for the SSG setting. The work connects online learning and adversarial game theory, demonstrating robust imitation-based learning in rich, information-constrained multi-agent environments and generalizing Feldman et al.'s copycat to Markov dynamics.

Abstract

Optimization under uncertainty is a fundamental problem in learning and decision-making, particularly in multi-agent systems. Previously, Feldman, Kalai, and Tennenholtz [2010] demonstrated the ability to efficiently compete in repeated symmetric two-player matrix games without observing payoffs, as long as the opponents actions are observed. In this paper, we introduce and formalize a new class of zero-sum symmetric Markov games, which extends the notion of symmetry from matrix games to the Markovian setting. We show that even without observing payoffs, a player who knows the transition dynamics and observes only the opponents sequence of actions can still compete against an adversary who may have complete knowledge of the game. We formalize three distinct notions of symmetry in this setting and show that, under these conditions, the learning problem can be reduced to an instance of online learning, enabling the player to asymptotically match the return of the opponent despite lacking payoff observations. Our algorithms apply to both matrix and Markov games, and run in polynomial time with respect to the size of the game and the number of episodes. Our work broadens the class of games in which robust learning is possible under severe informational disadvantage and deepens the connection between online learning and adversarial game theory.

Playing Markov Games Without Observing Payoffs

TL;DR

The paper tackles learning in zero-sum symmetric Markov games where payoff observations are unavailable and the opponent may have full knowledge of the game. It introduces three symmetry notions—per-state symmetric games (SSG), symmetry w.r.t. Markov policies (MSG), and symmetry w.r.t. history-dependent policies (HSG)—and shows deep structural reductions: MSG is equivalent to SSG via the EPR transformation, while HSG reduces to a single matrix game; these reductions yield tractable, polynomial-time strategies. The main results provide sublinear regret guarantees: in SSG (and thus MSG) theCopycat strategy achieves a bound of , and in HSG a bound of , with a matching lower bound for the SSG setting. The work connects online learning and adversarial game theory, demonstrating robust imitation-based learning in rich, information-constrained multi-agent environments and generalizing Feldman et al.'s copycat to Markov dynamics.

Abstract

Optimization under uncertainty is a fundamental problem in learning and decision-making, particularly in multi-agent systems. Previously, Feldman, Kalai, and Tennenholtz [2010] demonstrated the ability to efficiently compete in repeated symmetric two-player matrix games without observing payoffs, as long as the opponents actions are observed. In this paper, we introduce and formalize a new class of zero-sum symmetric Markov games, which extends the notion of symmetry from matrix games to the Markovian setting. We show that even without observing payoffs, a player who knows the transition dynamics and observes only the opponents sequence of actions can still compete against an adversary who may have complete knowledge of the game. We formalize three distinct notions of symmetry in this setting and show that, under these conditions, the learning problem can be reduced to an instance of online learning, enabling the player to asymptotically match the return of the opponent despite lacking payoff observations. Our algorithms apply to both matrix and Markov games, and run in polynomial time with respect to the size of the game and the number of episodes. Our work broadens the class of games in which robust learning is possible under severe informational disadvantage and deepens the connection between online learning and adversarial game theory.

Paper Structure

This paper contains 29 sections, 12 theorems, 75 equations, 2 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Symmetric Markov Games.
  4. Preliminaries
  5. Markov games
  6. The Copycat Strategy
  7. Problem definition and main results
  8. Our results
  9. Copycat strategy in SSG
  10. Strategy: Copycat on Each State
  11. General sum case
  12. Copycat strategy in MSG
  13. Expected Payoff Redistribution
  14. MSG is Equivalent to SSG
  15. Copycat strategy in HSG
  16. ...and 14 more sections

Key Result

Proposition 3.4

[proposition]prop:ssg In sec:SSG we show that the learner can guarantees $\abs{\mathbb{E}\brk[s]{\sum_{t =1}^T V ^{\pi _t ^1, \pi _t ^2}(s _1)}}=O(n \sqrt{T |S |H })$ for SSG games in $\mathrm{poly}(|S |, n , T )$ time.

Figures (2)

  • Figure 1: Example: SSG but not MSG game.
  • Figure 2: Example: MSG but not HSG game.

Theorems & Definitions (20)

  • Proposition 3.4
  • Proposition 3.5
  • Proposition 3.6
  • Proposition 3.7
  • Definition 5.1: Expected Payoff Redistribution; EPR
  • Lemma 5.2: EPR on MSG
  • Lemma 5.3
  • proof
  • Lemma 5.4
  • proof
  • ...and 10 more