Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Roping in Uncertainty: Robustness and Regularization in Markov Games

Jeremy McMahan, Giovanni Artiglio, Qiaomin Xie

TL;DR

It is shown that even for just reward-uncertain two-player zero-sum matrix games, computing an RNE is PPAD-hard and it is shown that RNE for two-player zero-sum RMG in this class can be provably solved in polynomial time.

Abstract

We study robust Markov games (RMG) with $s$-rectangular uncertainty. We show a general equivalence between computing a robust Nash equilibrium (RNE) of a $s$-rectangular RMG and computing a Nash equilibrium (NE) of an appropriately constructed regularized MG. The equivalence result yields a planning algorithm for solving $s$-rectangular RMGs, as well as provable robustness guarantees for policies computed using regularized methods. However, we show that even for just reward-uncertain two-player zero-sum matrix games, computing an RNE is PPAD-hard. Consequently, we derive a special uncertainty structure called efficient player-decomposability and show that RNE for two-player zero-sum RMG in this class can be provably solved in polynomial time. This class includes commonly used uncertainty sets such as $L_1$ and $L_\infty$ ball uncertainty sets.

Roping in Uncertainty: Robustness and Regularization in Markov Games

TL;DR

It is shown that even for just reward-uncertain two-player zero-sum matrix games, computing an RNE is PPAD-hard and it is shown that RNE for two-player zero-sum RMG in this class can be provably solved in polynomial time.

Abstract

We study robust Markov games (RMG) with -rectangular uncertainty. We show a general equivalence between computing a robust Nash equilibrium (RNE) of a -rectangular RMG and computing a Nash equilibrium (NE) of an appropriately constructed regularized MG. The equivalence result yields a planning algorithm for solving -rectangular RMGs, as well as provable robustness guarantees for policies computed using regularized methods. However, we show that even for just reward-uncertain two-player zero-sum matrix games, computing an RNE is PPAD-hard. Consequently, we derive a special uncertainty structure called efficient player-decomposability and show that RNE for two-player zero-sum RMG in this class can be provably solved in polynomial time. This class includes commonly used uncertainty sets such as and ball uncertainty sets.
Paper Structure (53 sections, 29 theorems, 88 equations)

This paper contains 53 sections, 29 theorems, 88 equations.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Related Work
  4. Robust MDPs.
  5. Robust MGs.
  6. Regularization in MDP and MGs.
  7. Notation.
  8. Preliminaries
  9. Robust Markov Game
  10. Robust solution concept.
  11. Existence of RNE.
  12. Regularized Markov Games
  13. Markov Games with Reward Uncertainty
  14. Matrix Games
  15. Reward Structure.
  16. ...and 38 more sections

Key Result

Theorem 2.1

(Existence of RNE). Given a RMG $(\mathcal{S},\{\mathcal{A}_{i}\}_{i\in[N]},P^{\star},\boldsymbol{r}^{\star},H,\mathcal{U})$ with $s$-rectangular uncertainty set $\mathcal{U}$ satisfying Assumption assu:reward, the robust Nash equilibrium defined in Definition (def:RNE) always exists. Moreover, a jo where $\textup{NE}(\cdot)$ denotes the Nash equilibrium of a general-sum, normal-form game.

Theorems & Definitions (61)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Theorem 2.1
  • Definition 2.4
  • Theorem 3.1
  • Corollary 3.2
  • Theorem 3.3
  • Corollary 3.4
  • Remark
  • ...and 51 more