Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Convex Optimal Value Functions For POSGs

Rafael F. Cunha, Jacopo Castellini, Johan Peralez, Jilles S. Dibangoye

TL;DR

This study mainly proves that the optimal state-value function is a convex function of occupancy states expressed on an appropriate basis in all zero-sum, common-payoff, and Stackelberg POSGs.

Abstract

Multi-agent planning and reinforcement learning can be challenging when agents cannot see the state of the world or communicate with each other due to communication costs, latency, or noise. Partially Observable Stochastic Games (POSGs) provide a mathematical framework for modelling such scenarios. This paper aims to improve the efficiency of planning and reinforcement learning algorithms for POSGs by identifying the underlying structure of optimal state-value functions. The approach involves reformulating the original game from the perspective of a trusted third party who plans on behalf of the agents simultaneously. From this viewpoint, the original POSGs can be viewed as Markov games where states are occupancy states, \ie posterior probability distributions over the hidden states of the world and the stream of actions and observations that agents have experienced so far. This study mainly proves that the optimal state-value function is a convex function of occupancy states expressed on an appropriate basis in all zero-sum, common-payoff, and Stackelberg POSGs.

On Convex Optimal Value Functions For POSGs

TL;DR

This study mainly proves that the optimal state-value function is a convex function of occupancy states expressed on an appropriate basis in all zero-sum, common-payoff, and Stackelberg POSGs.

Abstract

Multi-agent planning and reinforcement learning can be challenging when agents cannot see the state of the world or communicate with each other due to communication costs, latency, or noise. Partially Observable Stochastic Games (POSGs) provide a mathematical framework for modelling such scenarios. This paper aims to improve the efficiency of planning and reinforcement learning algorithms for POSGs by identifying the underlying structure of optimal state-value functions. The approach involves reformulating the original game from the perspective of a trusted third party who plans on behalf of the agents simultaneously. From this viewpoint, the original POSGs can be viewed as Markov games where states are occupancy states, \ie posterior probability distributions over the hidden states of the world and the stream of actions and observations that agents have experienced so far. This study mainly proves that the optimal state-value function is a convex function of occupancy states expressed on an appropriate basis in all zero-sum, common-payoff, and Stackelberg POSGs.
Paper Structure (20 sections, 28 theorems, 164 equations, 17 figures, 1 table)

This paper contains 20 sections, 28 theorems, 164 equations, 17 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background
  4. Notation.
  5. Master Games as POSGs
  6. Assumptions.
  7. Solution Concepts
  8. Slave Games As POMDPs
  9. Problem Reformulations
  10. Master Game Reformulations
  11. Slave Game Reformulations
  12. Connecting Slave and Master Games
  13. Sufficiency of Private Occupancy States
  14. Occupancy States As Mixtures of Private Occupancy States
  15. Convexity of Optimal Value Functions of Master Games
  16. ...and 5 more sections

Key Result

Lemma 1

The $t$-step state-value function $\upsilon^{i,a_{t:}}_{M,\gamma,t}\colon \mathcal{X}\times \mathcal{O}_t \to \mathbb{R}$ satisfies a recursion formula, known as bellman's equations, i.e., for any arbitrary hidden state $x_t$ and joint history $o_t$, with boundary condition $\upsilon^{i,\cdot}_{M,\gamma,\ell}(\cdot) = q^{i,\cdot}_{M,\gamma,\ell}(\cdot) \doteq 0$.

Figures (17)

  • Figure 1: The paper employs a three-step transformation methodology to convert original games into appropriate representations for optimal decision-making. The methodology involves the use of several game-theoretic models, including Partially Observable Stochastic Game (POSG), Partially Observable Markov Decision Process (POMDP), Markov Decision Process (MDP), Plan-Time Markov Game (PTMG), and Occupancy-State Markov Game (OMG). Best viewed in color.
  • Figure 2: Illustration of agents and their environment for the tiger problem.
  • Figure 3: A graphical model of a two-agent, partially observable stochastic game.
  • Figure 4: A $\ell$-step policy tree captures a sequence of $\ell$ time steps, each of which can be conditioned on the outcome of previous decisions. Each node is labelled with the decision made if it is reached.
  • Figure 5: Illustration of agents and their environment for the tiger problem where Calvin acts according to a public and fixed policy.
  • ...and 12 more figures

Theorems & Definitions (78)

  • Example 1
  • Definition 1
  • Definition 2
  • Definition 3
  • Lemma 1
  • proof
  • Definition 4
  • Definition 5
  • Definition 6
  • Definition 7
  • ...and 68 more