Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$ε$-Optimally Solving Zero-Sum POSGs

Erwan Escudie, Matthia Sabatelli, Jilles Dibangoye

TL;DR

This paper tackles the scalability challenge of solving zero-sum POSGs by embedding the game into an occupancy Markov game (OMG) and exploiting strong uniform continuity properties of the optimal value function. It introduces a two-component update operator that uses a linear program with an exponential reduction in constraints plus a polynomial-time update step, enabling efficient subgame solving and value improvement within a point-based iteration framework. The authors adapt point-based value iteration (PBVI) to offline planning in $M'$, provide rigorous error bounds, and develop pruning techniques to control the growth of value-function representations. Experiments on standard zs-POSG benchmarks demonstrate improved scalability to large horizons and applicability to several subclasses, outperforming or matching existing methods in many settings. This work lays a foundation for scalable dynamic programming and reinforcement learning approaches in zero-sum occupancy Markov games.

Abstract

A recent method for solving zero-sum partially observable stochastic games (zs-POSGs) embeds the original game into a new one called the occupancy Markov game. This reformulation allows applying Bellman's principle of optimality to solve zs-POSGs. However, improving a current solution requires solving a linear program with exponentially many potential constraints, which significantly restricts the scalability of this approach. This paper exploits the optimal value function's novel uniform continuity properties to overcome this limitation. We first construct a new operator that is computationally more efficient than the state-of-the-art update rules without compromising optimality. In particular, improving a current solution now involves a linear program with an exponential drop in constraints. We then also show that point-based value iteration algorithms utilizing our findings improve the scalability of existing methods while maintaining guarantees in various domains.

$ε$-Optimally Solving Zero-Sum POSGs

TL;DR

This paper tackles the scalability challenge of solving zero-sum POSGs by embedding the game into an occupancy Markov game (OMG) and exploiting strong uniform continuity properties of the optimal value function. It introduces a two-component update operator that uses a linear program with an exponential reduction in constraints plus a polynomial-time update step, enabling efficient subgame solving and value improvement within a point-based iteration framework. The authors adapt point-based value iteration (PBVI) to offline planning in $M'$, provide rigorous error bounds, and develop pruning techniques to control the growth of value-function representations. Experiments on standard zs-POSG benchmarks demonstrate improved scalability to large horizons and applicability to several subclasses, outperforming or matching existing methods in many settings. This work lays a foundation for scalable dynamic programming and reinforcement learning approaches in zero-sum occupancy Markov games.

Abstract

A recent method for solving zero-sum partially observable stochastic games (zs-POSGs) embeds the original game into a new one called the occupancy Markov game. This reformulation allows applying Bellman's principle of optimality to solve zs-POSGs. However, improving a current solution requires solving a linear program with exponentially many potential constraints, which significantly restricts the scalability of this approach. This paper exploits the optimal value function's novel uniform continuity properties to overcome this limitation. We first construct a new operator that is computationally more efficient than the state-of-the-art update rules without compromising optimality. In particular, improving a current solution now involves a linear program with an exponential drop in constraints. We then also show that point-based value iteration algorithms utilizing our findings improve the scalability of existing methods while maintaining guarantees in various domains.
Paper Structure (19 sections, 10 theorems, 18 equations, 8 figures, 3 tables, 2 algorithms)

This paper contains 19 sections, 10 theorems, 18 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Formulation
  4. Problem Reformulation
  5. Uniform Continuity Properties
  6. Exploiting Uniform Continuity
  7. Point-Based Value Iteration
  8. Experiments
  9. Conclusion
  10. Proof of Theorem \ref{['thm:greedy']}
  11. Proof of Theorem \ref{['thmsimstomgbellmanoptimalityeqn']}
  12. Proof of Theorem \ref{['thmerrorbound']}
  13. Pruning Methods
  14. Exact Pruning Method
  15. Bounded Pruning Method
  16. ...and 4 more sections

Key Result

Theorem 2.1

For any arbitrary $M'$, the optimal value functions $(\upsilon^*_0,\ldots, \upsilon^*_\ell)$ solutions of (eqn:bellman:optimality) are convex over marginal occupancy states for a fixed conditional occupancy-state family, i.e., there exists collections $(G_0,\ldots,G_\ell)$ of linear functions over where $g_{s^{\mathtt{c},\circ}_\tau}\colon O_\tau^{{\textcolor{gray}{\circ}}} \to \mathbb{R}$ is a

Figures (8)

  • Figure 1: Generalization across marginal occupancy states of the value function given by a collection $V = \{\textcolor{sthlmBlue}{g_{s^{\mathtt{c},\circ}}}, \textcolor{sthlmRed}{g_{s^{\mathtt{c},\circ}}}, \textcolor{sthlmGreen}{g_{s^{\mathtt{c},\circ}}}\}$ of linear functions over unknown marginal occupancy states. Figure A shows no generalization on marginal occupancy state $\textcolor{sthlmOrange}{\pmb{s^{\mathtt{m},\circ}}}$ because $\textcolor{sthlmOrange}{\pmb{s^{\mathtt{m},\circ}}} \notin \{\textcolor{sthlmBlue}{s^{\mathtt{m},\circ}}, \textcolor{sthlmRed}{s^{\mathtt{m},\circ}}, \textcolor{sthlmGreen}{s^{\mathtt{m},\circ}}\}$, cf. Theorem \ref{['thm:wiggers']}. Figure B shows generalization over unknown marginal occupancy state $\textcolor{sthlmOrange}{\pmb{s^{\mathtt{m},\circ}}}$ from known marginal occupancy state $\textcolor{sthlmBlue}{\pmb{s^{\mathtt{m},\circ}}}$ with offset $\kappa\|\textcolor{sthlmOrange}{s}-\textcolor{sthlmOrange}{s^{\mathtt{m},\circ}} \odot \textcolor{sthlmBlue}{s^{\mathtt{c},\circ}}\|_1$, cf. Theorem \ref{['thm:delage']}. Best viewed in color.
  • Figure 2: Generalization across occupancy states as provided by cunha2023convex's uniform continuity properties. Plot A describes generalization across all conditional occupancy states where the value function is given by a collection $\{\textcolor{sthlmBlue}{\omega}, \textcolor{sthlmRed}{\omega}, \textcolor{sthlmGreen}{\omega}\}$ of piecewise-linear and concave functions of conditional occupancy states, cf. Theorem \ref{['thm:cunha']}. Plot B describes generalization across any occupancy state $\textcolor{sthlmBlue}{\pmb{s}}$ given as a distribution over conditional occupancy states, such that value $\textcolor{sthlmBlue}{\omega(\pmb{s})}$ given by $\textcolor{sthlmOrange}{\pmb{s^{\mathtt{m},\circ}(o^{\circ})}} \cdot \textcolor{sthlmBlue}{\omega(\pmb{s^{\mathtt{c},o^{\circ}}})} + \textcolor{sthlmPurple}{\pmb{s^{\mathtt{m},\circ}(o^{\circ})}} \cdot\textcolor{sthlmBlue}{\omega(\pmb{s^{\mathtt{c},o^{\circ}}})} + \textcolor{sthlmYellow}{\pmb{s^{\mathtt{m},\circ}(o^{\circ})}} \cdot\textcolor{sthlmBlue}{\omega(\pmb{s^{\mathtt{c},o^{\circ}}})}$ is also a convex combinations of values from conditional occupancy states, cf. Theorem \ref{['thm:cunha:convex']}. In this form, the piece-wise linear and concave functions of conditional occupancy states become linear functions. Best viewed in color.
  • Figure 3: The linear program for the selection of greedy decision rule $\mathbb{G}^{{\textcolor{gray}{\bullet}}}(s_\tau,\upsilon^{{\textcolor{gray}{\bullet}}}_{\tau+1})$, with function $g_{V}^{\alpha} (o,u,z^{{\textcolor{gray}{\circ}}}) = \sum_x s_\tau(x,o) \sum_{y,z^{{\textcolor{gray}{\bullet}}}}p_{xy}^{uz}\cdot ( \frac{r_{xu}}{|Z^{{\textcolor{gray}{\circ}}}|} + \gamma\alpha(y,o^{{\textcolor{gray}{\bullet}}}u^{{\textcolor{gray}{\bullet}}}z^{{\textcolor{gray}{\bullet}}}))$. The red quantities are variables; green ones are constraint identifiers, and black ones are constants. Best viewed in color.
  • Figure 4: Computing $V_{\textcolor{sthlmRed}{\theta}}$ from the solution $\textcolor{sthlmRed}{\theta}$ of the linear program of Figure \ref{['fig:greedy:lp']}. Best viewed in color.
  • Figure 5: A visual representation of the performance of our proposed algorithms for the challenging horizon $\ell = 10$ across five different games. We can see that for Adversarial Tiger, Competitive Tiger and Recycling all three PBVI variants perform equally. PBVI$_2$ also matches the performance of PBVI$_1$ on Mabc and Matching Pennies, with the latter benchmark being harder to solve for PBVI$_3$. Best viewed in color.
  • ...and 3 more figures

Theorems & Definitions (13)

  • Theorem 2.1: Adapted from Wiggers16
  • Theorem 2.2: Adapted from DelBufDibSaf-DGAA-23
  • Theorem 2.3: Adapted from cunha2023convex
  • Theorem 2.4: Adapted from cunha2023convex
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 3
  • proof
  • Theorem 3
  • ...and 3 more